How we do user research

Introduction

This guide explains how we do user research at dxw.

The guidance applies to all user researchers employed by dxw, both staff and contractors.

The guidance builds on our user research principles. It also builds on the guidance for user research in the GDS Service Manual so we can help teams meet the Government Service Standard.

Some of the guidance explains how to do user research in an effective, ethical and legal way. If your situation isn’t covered, and you’re not sure what to do, ask Coca Rivas, Director of Design, or John Waterworth, Principal Researcher.

User research workflow

These are the things that user researchers usually do on projects.

  1. Support sales and scheduling

    Researchers contribute to client proposals and scheduling projects. We advise on research activities to include in proposals, join in pitch presentations and help with scheduling decisions.

    This helps dxw to win good work that delivers valuable results for our clients, for users and for ourselves.

  2. Help setup the project and team

    Researchers get involved with project inception and planning. We review existing evidence. And we create research plans that show what we’ll do, why and how we’ll do it. We make sure the plan is achievable and allows us to do research at a sustainable pace.

    This helps teams to focus on understanding and meeting the needs of users, and to produce the best results with the people, time and resources we have.

    How we create and use research plans

  3. Recruit people for research

    Researchers recruit appropriate participants for research activities. Finding the right participants at the right time is always challenging, so we start recruiting as soon as we can and use a variety of approaches to reach everyone we need. We keep track of who participates in our research, so we can be confident that we’re including the right people.

    This helps teams to learn about all their users and produce services that work well for everyone.

    How we get informed consent for user research

    When and how we give incentives to research participants

    How we protect participant privacy

  4. Plan and do research activities

    This is the most obvious and visible part of our work. We choose research activities that provide strong evidence and reliable answers, for the least time, effort and cost. And we choose activities that help our teams and different kinds of users to participate.

    We record what we do and how we do it, for each other, and for those who come after us.

    This helps teams to embed regular research into their agile practices and get the best results from the effort they put in.

    How we create and use research plans

    How we write and use discussion guides

    How we take notes and record research sessions

  5. Analyse research and produce findings

    Researchers carefully analyse the different kinds of data we collect. We analyse data in batches and record our findings from sprint to sprint. And we involve our colleagues and clients in analysis to make sure our findings are clear and relevant.

    This helps teams understand, believe in and act quickly on what they learn.

  6. Help teams make good decisions

    Researchers produce findings and other recommendations that help teams make good decisions about what to do next and how to approach it. We get actively involved in identifying opportunities, creating design ideas and setting priorities.

    This helps teams to create great services that work well for users, and to strike a balance between short-term wins and more complex, long term changes.

  7. Create and share outputs

    Researchers create and share formal outputs like plans, presentations, reports and diagrams. These record and communicate the important things we learn. We create finished outputs from sprint to sprint, not just at the end of a project. And we think carefully about how we present and share these outputs to produce the most impact.

    This helps our clients and other stakeholders use and benefit from what we learned both during the project and long after we’ve gone.

    How we share research activities and findings

  8. Take care of research data

    Researchers know it’s important to manage research data carefully. We follow our privacy policy and ISO27001 certification when collecting, processing and storing research data to make sure we comply with that General Data Protection Regulation.

    This helps teams to get the most from the research data they collect, while protecting the privacy of the people taking part.

    How we take notes and record research sessions

    How we take care of research data

How we name and identify documents

It’s important that team members and stakeholders know they have the right research plan, discussion guide or other research document.

So we give our documents consistent file names like:

  • YYYY MM DD - TYPE - CLIENT - PROJECT - RESEARCH ACTIVITY (if needed for clarity)

For example:

  • 2019 10 01 - Research plan - Southwark - Affordable homes monitoring alpha

And a consistent front cover with:

  • Client and dxw logos
  • Client and project names
  • Document type
  • Date
  • Main contact (who wrote it)

For example:

  • Southwark Council
  • Affordable homes monitoring alpha
  • User research plan
  • (Draft)
  • 1 October 2019
  • John Waterworth

How we create and use research plans

Planning research activities

It’s important that we carefully plan the research we do.

Teams create the best services when research activities provide reliable answers to their most important questions. And we do our best research when we do it in ethical, inclusive and sustainable ways.

We want to involve our colleagues and other stakeholders in creating research plans, so they understand and can contribute to research activities.

And we want to create plans that allow us to learn, share and adapt as we go.

The guide to planning user research for your service in the Service Manual sets out a good approach to creating a plan.

Research plan template

Creating and sharing a short research plan document is a great way to get everyone aligned. And to address any concerns that clients and stakeholders may have.

We have a research plan template that you can use as a starting point. But feel free to adapt it and add sections that may be important for your project.

How we do research safely during coronavirus

We have a responsibility to keep ourselves and our participants safe and comply with government guidance during the coronavirus (COVID-19) outbreak. In particular, we should carefully consider whether and how to do any face to face research.

For now, you will need the approval of a Director, Head or Principal Researcher to do face to face research. And you must document your decision and your assessment of the risks in your research plan.

The Service Manual has a guide to choosing face to face or remote research, and what to include when you document your decision and your assessment of the risks.

How we write and use discussion guides

We create discussion guides to help us plan and do research activities, like interviews, visits, workshops and usability tests.

They help us to:

  • identify the purpose of the research activity and what we want to get from it
  • review planned activities with team members and stakeholders so we stay aligned with project goals
  • stay on track during research sessions
  • make sure different researchers cover the same topics so participants have a consistent experience
  • keep a record of what we did in each research activity

The parts of a discussion guide

Introduction

Recap the goals of the project and the research activity, describe the methods you’ll be using, and set out the overall structure of the research session.

Preparation

Describe anything the researcher needs to have or do, to prepare for the session. Reference things like the information sheet to send to participants, describe how to set up the prototype you’ll be testing, and reference the notes template you want to use.

Welcome

Set out the things you’ll say to the participant to let them them know what’s going on and make sure they feel comfortable. It can be helpful to write this out word for word, so you have something to fall back on when you’re tired, or get interrupted or distracted.

In your welcome you should:

  • Say thank you - Start off by letting the participant know you appreciate them offering their time (incentivised or not)
  • Introduce yourself - Let them know your name, role, company and who you are working for
  • State your relationship to the client - We are independent researchers, working on behalf of a client
  • Summarise the research purpose - Explain what you’re trying to find out, eg We’re interested in how you recruit teachers for your school
  • Remind them it is voluntary - They can choose to stop at any time
  • Reassure tham that there are no right or wrong answers - We’re interested in how the participant thinks and what influences that
  • Ask them to be honest - As researchers we won’t be offended by any comments, we just want to find out what people really think about a service and how it might be improved
  • Describe how the session will go - Let the participant know who is taking notes or observing, how you are recording the session, how long the session will take, and what you’ll be doing in that time, so the participant knows what to expect
  • Check consent and non-disclosure - Has this been done? Do they have any questions?

Topics, tasks and activities

Have a section in your guide for each of the main interview topics, test tasks or workshop activities.

For each one write out:

  • What you are trying to learn
  • For an interview topic, the starter questions you’ll ask along with likely follow ups
  • For a test task or workshop activity
    • the steps to do
    • explanations and instructions for participants, such asking them to think out loud during a test so you can understand what’s happening
  • How long the topic, task or activity is likely to take
  • The priority - is this essential, or could you leave it out if you’re short of time

Wrap up

Describe how the researcher should conclude the session with the participant. This can include asking the participant for their final thoughts, asking them how the session went for them, letting them know what will happen next, and thanking them for their time.

Also include any steps for the researcher, like collecting and storing recordings, or deleting data from prototypes.

Three discussion guide templates

We have template discussion guides for three different kinds of research activities:

How we share research activities and findings

We regularly share descriptions of research activities and findings in show and tells, reports and other places.

Slide layouts for research

When presenting about research, we use one of the dxw Proposal Deck Templates. These have all the kinds of slide layouts you will need.

You can find these in the Google Slides Template Gallery.

Presenting research activities

When sharing what we plan to do or what we’ve been doing, we talk about research activities rather than research methods.

Descriptions of research activities look like:

We did some kind of research with some kinds of participants to learn something the team needs to know.

We avoid using research jargon that our colleagues might not understand. For example, rather than talking about ‘ethnography’ or ‘contextual enquiry’ we might say that “we visited the support desk and observed the team helping people with their applications”.

Presenting research findings

When sharing findings, we always make the finding the headline. Then follow up with supporting evidence, explanation and illustration.

You can read more about this in the Sharing user research findings guide in the Service Manual.

You must have informed consent from participants before you start any research activity with them. And you must then act only within the consent the participant has given.

This guidance explains how we collect and keep evidence of informed consent at dxw. It builds on the general guidance on getting informed consent in the Service Manual.

There are two basic options:

  1. The simplest and best option is to use our client’s existing consent materials. If they don’t have any, we can work with the client to create consent materials for the project, that the client can then reuse in future.

    In this case the client will manage all the research data we collect and keep the associated records of consent.

  2. If using the client’s consent materials is not possible, or if the client asks us to do confidential research, we can use the dxw information sheet and consent form templates.

    In this case dxw will manage the research data and keep the associated records of consent.

Which ever approach you take, make sure your consent materials and process follow the good practice for informed consent described in the Service Manual.

Creating an information sheet

For each round or batch of research you do, you should create a specific information sheet that describes the research you are doing, and gives the participant the information they need.

You can use the same information sheet for several rounds or batches of research, as long as you are:

  • doing the research for the same purpose
  • doing the same things during the sessions
  • collecting the same research data
  • working with the same third-party organisations and suppliers

We have template information sheets to use for the most common research situations.

If you are using a client’s consent form, make sure it clearly describes dxw’s involvement in the research, and how we will share research data and findings.

We have template consent forms for the most common research situations.

If none of the standard consent forms will work for the research activities you are doing, speak to the Head of User Research.

Some clients and stakeholders will not understand the ethical and legal reasons for getting informed consent, and will not be familiar with good consent practice.

To avoid problems later, share your consent materials and process as soon as you can with the client and with any organisations whose staff or members you are researching with.

Providing the information sheet

Give each participant a copy of the information sheet when they are being recruited for a research session. Check which format and channel works best for them - email attachment, paper letter, easy read, etc.

For a face to face research session, make sure you can provide a paper copy of both the information sheet and the associated privacy statement/privacy policy, if a participant asks for one.

Face to face research sessions

For user research sessions where you meet the participant in person, you can:

  • give the participant a printed copy of the consent form
  • send the consent form in an accessible digital format the participant can read using their assistive technology
  • read the consent form aloud

Check with the participant which format works best for them.

Give the participant the consent form as soon as you can. Make sure they have time to read it carefully, and an opportunity to ask questions, before you ask the participant to sign .

Remote or telephone research sessions

For user research sessions where you will not meet the participant face to face, you can:

  • attach the consent form to an email in an accessible digital format
  • send a paper consent form in the post

Ask the participant to send a reply, confirming their consent. They can reply to the email, or post back a signed paper consent form.

During the session

At the start of the session, remind the participant that their participation is voluntary and they can stop or take a break at any time. Remind the participant about the different ways you are recording the session, and about anyone observing or listening in. Ask the participant to confirm their consent for the recording.

As the session continues, actively make sure you have the participant’s consent. For example, check in with the participant if they look concerned about an activity, or if you’re on a visit and want to take a photo in a new area.

For paper consent forms, scan or photograph the signed consent form and keep the image file in the folder with the research data you collect in the session.

For email responses, keep a copy of the consent form and of the participant’s reply in the folder with the research data you collect in the session.

Make sure the consent form, email reply and any research data have the same participant identifier. Do not include participant names or other personal identifiers in the document names.

When and how we give incentives to research participants

It‘s normal to pay members of the public an incentive when they take part in research activities with us.

This guidance explains how we agree incentives with clients and pay incentives to participants.

It builds on the basic guidance on giving incentives in the Service Manual.

Who gets incentives

We will normally pay incentives to members of the public, including people who use a government service for their work. Professionals and members of groups may not be comfortable with getting an incentive themselves, and may want the incentive given to their organisation or donated to charity.

We do not expect to pay incentives to our clients‘ staff or to other public servants who take part in research related to their work.

When we are paying an incentive, make sure the participant knows the amount of the incentive and how they will receive it. Include this in the information sheet.

Agreeing incentives

It’s best to include estimates of incentives and other research expenses in the statement of work.

You can then confirm the exact incentive with the client when you are planning a batch of research.

On previous project we have paid:

  • £30 of Love2Shop vouchers for 1.5 hours of in-person usability testing
  • £50 cash for geographically-specific users to take part in 1 hour of remote mobile usability testing
  • £100 cash for an hour-long discovery research session with people likely to need assisted digital support
  • £100 cash for 45 minute research session with small businesses

Confirming incentives at the start of the research session

Handle the incentive at the start of the research session - either giving the incentive, or confirming that the incentive will be sent. This avoids any suggestion that participants have to ‘earn’ their incentive. It also means you don’t have to deal with the incentive if you want to stop a problematic session.

Paying incentives through a recruitment agency

If you are using a recruitment agency, ask them to pay the incentive to the participant. They will know how they do this. They may charge a small fee for handling the incentive.

If a participant contacts dxw to say they haven’t received their incentive, apologise and contact the agency to let them know.

Paying incentives using shopping vouchers

Where we are paying the incentive, we use shopping vouchers. Normally from Love2shop.

Request the amount of vouchers you need through the #help-purchasing Slack channel and purchasing form. They should be in £5 and £10 denominations so they are easier to spend.

In a face-to-face research session, give the vouchers to the participant and have them sign a receipt.

For a remote research session, ask for the vouchers to be posted or emailed to the participant. When posting vouchers, enclose the vouchers with a signed thank you sheet.

Other ways to pay incentives

Other options for paying incentives might be a donation to a preferred charity, or a hamper that can be shared with a team.

Withholding incentives

We do not pay the incentive (or any recruitment fee) if the participant doesn’t show up for their session. Although we might still pay the incentive if the participant tries to attend, but is stopped by an emergency or severe travel problem beyond their control.

If a recruitment agency provides a participant who does not match the brief, we will pay the participant their incentive, but we will not pay the agency their recruitment fee.

How we take notes and record research sessions

To capture what happens in our research sessions we take notes and photos, make audio, video and screen recordings, and take copies of paperwork and other material that participants use or refer to.

We use these notes, photos and recordings during analysis to make sure we produce valid findings. And we use extracted quotes, images and clips to illustrate the findings we share.

There is good guidance on Taking notes and recording user research sessions in the Service Manual, and on Capturing User Research in UX Matters.

Here we provide additional, specific guidance for researchers at dxw.

Good practices to follow

  1. Capture only what you need and have consent for

    Good user research practice, and good privacy practice, is to capture only the data you believe you need. For example, don’t make a screen or video recording, ‘just in case’, if you’ll only ever need a transcript of what the participant says.

    Make sure that anything you capture during a session is covered by the consent you’ve collected. If a participant changes their mind about what they consent to, be prepared to not make, or to delete recordings you no longer have consent for.

  2. Do not use your personal devices

    Always use a dxw provided device, rather than your personal device, when taking notes and photos, or making recordings.

    Using our own devices in research means that personal data about research participants can be mixed into our personal data stores, like cloud photo libraries. And also runs the risk of participants, client staff and colleagues seeing our personal messages and images.

  3. Keep your notes and recordings secure

    As soon as possible after the session, transfer any notes and recordings from paper and other devices to your dxw provided laptop. And from your laptop to the dxw Google Drive or to the client’s research store.

    Keep only anonymised notes, quotes and images in tools like Miro and Trello.

  4. Have a fall back

    The one certainty in user research is that things go wrong. So make sure you have other options to capture what you need.

    For example:

    • The participant may decide they don’t want to be recorded in a particular way, so be prepared to switch from video to an audio-only recording, or to just take notes
    • Your preferred device, app or service may not be working properly, so have an alternative like a paper notebook
    • Your note taker, assistant or observer may not be able to join the session, so be prepared to take notes and recordings yourself

    Note that if the participant refuses all recording, including note taking, you may want to abandon the session.

Using your dxw provided laptop

Whenever possible, use your dxw provided laptop for taking notes and making recordings. Our laptops have encrypted drives and we follow good security policies.

With your laptop you can:

  • capture notes to a local file or a Google Sheet
  • record audio using a microphone and the Voice Memos or Quicktime apps
  • record video using an external camera and Quicktime
  • record your laptop screen using Quicktime or Screenshot

If research participants might use your laptop or see your screen during a research session, it’s a good idea to create a separate ‘research’ user account on your laptop with things like notifications and browser autofill turned off.

Using a dxw provided handheld device

Use a dxw provided handheld device, rather than own phone or tablet, to take photos and short videos during interviews, visits and workshops.

Remember to delete the photos and recordings from the device once you have transfered them to your laptop or to Google Drive.

Using the notes template

We have a template for a Google Sheet you can use to take notes during a research session.

You can make a copy of the sheet for each research session, or make a copy for each batch of research and create tabs within the sheet for each research session.

Using sticky notes and paper worksheets

We often use sticky notes and paper worksheets to capture information during research sessions, for example when researching users’ experiences.

Design paper worksheets, and give people advice on writing sticky notes, so you can easily use the information recorded.

Avoid capturing personally identifiable information about participants on sticky notes or paper. If you capture any sensitive informaton on sticky notes or paper, make sure to use a shredder to dispose of it.

Use a dxw provided phone or tablet, rather than your personal device, to capture the contents of sticky notes and paper worksheets.

Using a video conferencing service

Our preferred service for recording remote screen sharing sessions is GoToMeeting. Setup GoToMeeting so that the recorded video is saved onto your dxw provided laptop. And then transfer the video to Google Drive as soon as possible.

Depending on the client, we may need to use their preferred video conferencing service. If the service stores recordings ‘in the cloud’, make sure to download the video as soon as possible and delete it from the service.

Using third-party facilities

If you are using a third party facility like a usability lab, transfer any recordings made by the lab to your dxw provided laptop as soon as possible after the sessions are complete.

How we protect participant privacy

When we do research, we must carefully manage participants’ contact details and other personal data to protect their privacy and comply with the law.

This guidance explains how we collect and use information about research participants at dxw. It builds on the general guidance on managing participant privacy in the Service Manual.

Collecting only the participant details we need

We collect as little information about participants as possible. And only the information we really need to manage their participation. For example, collecting just the participant’s name and email address to arrange a video call.

When we use a screening questionnaire, we ask only the questions we need to select participants. For example, we don’t ask for a date of birth if we only need an age range.

We avoid keeping participants’ personal details as a record of who we spoke to, or to manage subject access or deletion requests. We may keep counts of numbers and types of participants, but these should not include any personal details.

Storing participant details securely

We usually store participants’ details in the Google Drive folders for the relevant project, for example in a spreadsheet in a Research subfolder. We limit access to just the colleagues who need to use them to help people participate.

The Google Drive Help Center has a useful guide on ways to stop, limit, or change sharing of files and folders.

When using an online questionnaire tool to recruit and screen participants, transfer the responses from the tool to Google Drive as soon as possible, and securely delete the original responses from the tool.

When sending calendar invites to participants, we restrict access to the guest list so that only colleagues involved in the research will see participants’ contact details.

We avoid sharing email conversations that include participants’ contact and other personal details.

And we avoid sharing participants’ details in any other tool, such as Slack or Trello.

Deleting participant details when no longer needed

We delete participants’ details from Google Drive, Gmail and Calendar as soon as we no longer need the details to manage their participation in research.

Labelling emails exchanged with participants is a good way to find and delete them. For example, in Google Gmail you might create ‘Participant’ and ‘RODA’ labels and search for them using ‘label:(participant RODA)’.

Identifying calendar events that might contain participants’ details is not so easy. In Google Calendar you can create a personal Research calendar and add all research activities involving participants to that calendar. Or you might adopt a naming convention such as ‘Research:’.

Doing fully confidential research

We sometimes do research where the participants are known only to the researchers. Other members of the team will not know who participated in the research.

When doing fully confidential research:

  • we strictly limit access to participant details
  • use private meeting invites
  • share only fully anonymised findings

How we take care of research data

When we do research, we must manage the data we collect so that we protect participants’ privacy and comply with the law.

This guidance explains how we manage research data at dxw. It builds on the general guidance on managing research data in the Service Manual. These is also useful advice in The User Researcher’s Guide to GDPR.

Agreeing who controls the research data

There are two basic options:

  1. The client is the data controller for the research data, and we are a processor.

    This is the best approach when we are working in a blended team for a client with an established user research practice. It allows the client and team to continue accessing and using the research data independent of dxw.

    In this case we use the client’s consent materials and store the research data in their systems.

  2. We are the data controller for the research data, and we limit the client’s access to the research data.

    This is the best approach when we are working in a dxw staffed team for a client with a limited or no user research practice. It reduces the burden on the client and reduces our dependence on their practices.

    In this case we use dxw consent materials and store the research data in our systems along with the records of consent.

We will agree on the approach for the project as early as possible. Ideally as part of the statement of work, but at the latest during inception.

Storing research data securely

We store research data that we control in the Client work area of the dxw Google Drive.

We limit access to folders and files containing research data to colleagues who need to access and use the data to complete their work.

The Google Drive Help Center has a useful guide on ways to stop, limit, or change sharing of files and folders.

When we capture research data on a device or in another online service, we move the data to Google Drive as soon as possible, and securely delete the research data from the device or service.

If moving research data to Google Drive is not immediately possible, we move the data to our dxw provided laptop. For example, if we collect videos during pop-up research at a location with no reliable Internet connection.

We store records of consent along with the research data they relate to.

The consent record might be a scan of a paper consent form, a saved copy of an email, or some other suitable record.

We delete the consent record when we have deleted that participant’s research data and contact details.

Identifying research data clearly

To identify research data in Google Drive, we follow our general file naming convention.

  • We keep research data in subfolders identified by the date of the research, the project and the research activity. For example:

    2021 04 - Research - BEIS ODA - Observing the ODA QA process

  • We name individual files with the date collected, the project and the research activity. For example:

    2021 04 - Notes - BEIS ODA - Observing the ODA QA process

  • If needed, we add participant numbers to identify the different data files within a research activity. For example:

    2021 04 08 - P2 - Recording - BEIS ODA - Observing the ODA QA process

  • We use matching file names to keep records of consent. For example:

    2021 04 08 - P2 - Consent - BEIS ODA - Observing the ODA QA process

Be careful not to use a participant’s personal details, such as their name or organisation, to identify the data we collect during research activities.

Sharing anonymised extracts

We often use extracts from research data to illustrate findings and other project outputs. These extracts might be quotes, photos, video clips, screenshots, or copies of artefacts like documents.

In any material that may go outside of the immediate team, we use only fully anonymised extracts, where the participant cannot be identified.

The UK Data Service has useful guidance on anonymising both quantitative and qualitative research data.

We sometimes do show identifiable extracts within a team, and where we have consent. For example, showing clips from a usability testing session during a closed playback with the team.

Deleting research data when no longer needed

We delete research data from Google Drive as soon as we no longer need it for our work. Or at the latest by the end of the declared retention period, usually 1 year.

Research data files may be owned by different members of staff. To delete files owned by colleagues, including those who have left dxw, contact the technical operations team at #help-internal-tech-support.

Participants can withdraw their consent or ask us to share any personal data we have about them.

If a participant withdraws their consent, or if it is not clear that we have a participant’s consent, then we must find and delete any personal data we have, including notes, recordings, emails, calendar events, records in spreadsheets, etc.

If a participant asks to see the personal data we have about them, then we must find and share any personal data we have, including notes, recordings, emails, calendar events, records in spreadsheets, etc.

To identify the participant’s data, we may need to ask them for details of the research activities they were involved in.

And we may need to separate data about one participant from data about other participants. For example, if we have notes or voice recordings from a group workshop.