How we do user research

Introduction

This guide explains how we do user research at dxw digital.

The guidance applies to all user researchers employed by dxw digital, both staff and contractors.

The guidance builds on our user research principles. It also builds on the guidance for user research in the GDS Service Manual so we can help teams meet the Government Service Standard.

Some of the guidance explains how to do user research in an effective, ethical and legal way. If your situation isn’t covered, and you’re not sure what to do, ask John Waterworth, the Head of User Research.

User research workflow

These are the things that user researchers usually do on projects.

  1. Support sales and scheduling

    Researchers contribute to client proposals and scheduling projects. We advise on research activities to include in proposals, join in pitch presentations and help with scheduling decisions.

    This helps dxw digital to win good work that delivers valuable results for our clients, for users and for ourselves.

  2. Help setup the project and team

    Researchers get involved with project inception and planning. We review existing evidence. And we create research plans that show what we’ll do, why and how we’ll do it. We make sure the plan is achievable and allows us to do research at a sustainable pace.

    This helps teams to focus on understanding and meeting the needs of users, and to produce the best results with the people, time and resources we have.

  3. Recruit people for research

    Researchers recruit appropriate participants for research activities. Finding the right participants at the right time is always challenging, so we start recruiting as soon as we can and use a variety of approaches to reach everyone we need. We keep track of who participates in our research, so we can be confident that we’re including the right people.

    This helps teams to learn about all their users and produce services that work well for everyone.

    How we get informed consent for user research

    When and how we give incentives to research participants

  4. Plan and do research activities

    This is the most obvious and visible part of our work. We choose research activities that provide strong evidence and reliable answers, for the least time, effort and cost. And we choose activities that help our teams and different kinds of users to participate.

    We record what we do and how we do it, for each other, and for those who come after us.

    This helps teams to embed regular research into their agile practices and get the best results from the effort they put in.

    How we write and use discussion guides

    How we take notes and record research sessions

  5. Analyse research and produce findings

    Researchers carefully analyse the different kinds of data we collect. We analyse data in batches and record our findings from sprint to sprint. And we involve our colleagues and clients in analysis to make sure our findings are clear and relevant.

    This helps teams understand, believe in and act quickly on what they learn.

  6. Help teams make good decisions

    Researchers produce findings and other recommendations that help teams make good decisions about what to do next and how to approach it. We get actively involved in identifying opportunities, creating design ideas and setting priorities.

    This helps teams to create great services that work well for users, and to strike a balance between short-term wins and more complex, long term changes.

  7. Create and share outputs

    Researchers create and share formal outputs like plans, presentations, reports and diagrams. These record and communicate the important things we learn. We create finished outputs from sprint to sprint, not just at the end of a project. And we think carefully about how we present and share these outputs to produce the most impact.

    This helps our clients and other stakeholders use and benefit from what we learned both during the project and long after we’ve gone.

  8. Take care of research data

    Researchers know it’s important to manage research data carefully. We follow our privacy policy and ISO27001 certification when collecting, processing and storing research data to make sure we comply with that General Data Protection Regulation.

    This helps teams to get the most from the research data they collect, while protecting the privacy of the people taking part.

    How we take notes and record research sessions

How we name and identify documents

It’s important that team members and stakeholders know they have the right research plan, discussion guide or other research document.

So we give our documents consistent file names like:

  • YYYY MM DD - TYPE - CLIENT - PROJECT - RESEARCH ACTIVITY (if needed for clarity)
  • For example, 2019 10 01 - User research plan - Southwark - Affordable homes monitoring alpha

And a consistent front cover with:

  • Client and dxw logos
  • Client and project names
  • Document type
  • Date
  • Main contact (who wrote it)

For example:

  • Southwark Council
  • Affordable homes monitoring alpha
  • User research plan
  • (Draft)
  • 1 October 2019
  • John Waterworth

How we write and use discussion guides

We create discussion guides to help us plan and do research activities, like interviews, visits, workshops and usability tests.

They help us to:

  • identify the purpose of the research activity and what we want to get from it
  • review planned activities with team members and stakeholders so we stay aligned with project goals
  • stay on track during research sessions
  • make sure different researchers cover the same topics so participants have a consistent experience
  • keep a record of what we did in each research activity

The parts of a discussion guide

Introduction

Recap the goals of the project and the research activity, describe the methods you’ll be using, and set out the overall structure of the research session.

Preparation

Describe anything the researcher needs to have or do, to prepare for the session. Reference things like the information sheet to send to participants, describe how to set up the prototype you’ll be testing, and reference the notes template you want to use.

Welcome

Set out the things you’ll say to the participant to let them them know what’s going on and make sure they feel comfortable. It can be helpful to write this out word for word, so you have something to fall back on when you’re tired, or get interrupted or distracted.

In your welcome you should:

  • Say thank you - Start off by letting the participant know you appreciate them offering their time (incentivised or not)
  • Introduce yourself - Let them know your name, role, company and who you are working for
  • State your relationship to the client - We are independent researchers, working on behalf of a client
  • Summarise the research purpose - Explain what you’re trying to find out, eg We’re interested in how you recruit teachers for your school
  • Remind them it is voluntary - They can choose to stop at any time
  • Reassure tham that there are no right or wrong answers - We’re interested in how the participant thinks and what influences that
  • Ask them to be honest - As researchers we won’t be offended by any comments, we just want to find out what people really think about a service and how it might be improved
  • Describe how the session will go - Let the participant know who is taking notes or observing, how you are recording the session, how long the session will take, and what you’ll be doing in that time, so the participant knows what to expect
  • Check consent and non-disclosure - Has this been done? Do they have any questions?

Topics, tasks and activities

Have a section in your guide for each of the main interview topics, test tasks or workshop activities.

For each one write out:

  • What you are trying to learn
  • For an interview topic, the starter questions you’ll ask along with likely follow ups
  • For a test task or workshop activity
    • the steps to do
    • explanations and instructions for participants, such asking them to think out loud during a test so you can understand what’s happening
  • How long the topic, task or activity is likely to take
  • The priority - is this essential, or could you leave it out if you’re short of time

Wrap up

Describe how the researcher should conclude the session with the participant. This can include asking the participant for their final thoughts, asking them how the session went for them, letting them know what will happen next, and thanking them for their time.

Also include any steps for the researcher, like collecting and storing recordings, or deleting data from prototypes.

Three discussion guide templates

We have template discussion guides for three different kinds of research activities:

You must have informed consent from participants before you start any research activity with them. And you must then act only within the consent the participant has given.

This guidance explains how we collect and keep evidence of informed consent at dxw digital. It builds on the general guidance on getting informed consent in the Service Manual.

There are two basic options:

  1. The simplest and best option is to use our client’s existing consent materials. If they don’t have any, we can work with the client to create consent materials for the project, that the client can then reuse in future.

    In this case the client will manage all the research data we collect and keep the associated records of consent.

  2. If using the client’s consent materials is not possible, or if the client asks us to do confidential research, we can use the dxw digital information sheet and consent form templates.

    In this case dxw will manage the research data and keep the associated records of consent.

Which ever approach you take, make sure your consent materials and process follow the good practice for informed consent described in the Service Manual.

Creating an information sheet

For each round or batch of research you do, you should create a specific information sheet that describes the research you are doing, and gives the participant the information they need.

You can use the same information sheet for several rounds or batches of research, as long as you are:

  • doing the research for the same purpose
  • doing the same things during the sessions
  • collecting the same research data
  • working with the same third-party organisations and suppliers

We have template information sheets to use for the most common research situations.

If you are using a client’s consent form, make sure it clearly describes dxw digital’s involvement in the research, and how we will share research data and findings.

We have template consent forms for the most common research situations.

If none of the standard consent forms will work for the research activities you are doing, speak to the Head of User Research.

Some clients and stakeholders will not understand the ethical and legal reasons for getting informed consent, and will not be familiar with good consent practice.

To avoid problems later, share your consent materials and process as soon as you can with the client and with any organisations whose staff or members you are researching with.

Providing the information sheet

Give each participant a copy of the information sheet when they are being recruited for a research session. Check which format and channel works best for them - email attachment, paper letter, easy read, etc.

For a face to face research session, make sure you can provide a paper copy of both the information sheet and the associated privacy statement/privacy policy, if a participant asks for one.

Face to face research sessions

For user research sessions where you meet the participant in person, you can:

  • give the participant a printed copy of the consent form
  • send the consent form in an accessible digital format the participant can read using their assistive technology
  • read the consent form aloud

Check with the participant which format works best for them.

Give the participant the consent form as soon as you can. Make sure they have time to read it carefully, and an opportunity to ask questions, before you ask the participant to sign .

Remote or telephone research sessions

For user research sessions where you will not meet the participant face to face, you can:

  • attach the consent form to an email in an accessible digital format
  • send a paper consent form in the post

Ask the participant to send a reply, confirming their consent. They can reply to the email, or post back a signed paper consent form.

During the session

At the start of the session, remind the participant that their participation is voluntary and they can stop or take a break at any time. Remind the participant about the different ways you are recording the session, and about anyone observing or listening in. Ask the participant to confirm their consent for the recording.

As the session continues, actively make sure you have the participant’s consent. For example, check in with the participant if they look concerned about an activity, or if you’re on a visit and want to take a photo in a new area.

For paper consent forms, scan or photograph the signed consent form and keep the image file in the folder with the research data you collect in the session.

For email responses, keep a copy of the consent form and of the participant’s reply in the folder with the research data you collect in the session.

Make sure the consent form, email reply and any research data have the same participant identifier. Do not include participant names or other personal identifiers in the document names.

When and how we give incentives to research participants

It‘s normal to pay members of the public an incentive when they take part in research activities with us.

This guidance explains how we agree incentives with clients and pay incentives to participants.

It builds on the basic guidance on giving incentives in the Service Manual.

Who gets incentives

We will normally pay incentives to members of the public, including people who use a government service for their work. Professionals and members of groups may not be comfortable with getting an incentive themselves, and may want the incentive given to their organisation or donated to charity.

We do not expect to pay incentives to our clients‘ staff or to other public servants who take part in research related to their work.

When we are paying an incentive, make sure the participant knows the amount of the incentive and how they will receive it. Include this in the information sheet.

Agreeing incentives

It’s best to include estimates of incentives and other research expenses in the statement of work.

You can then confirm the exact incentive with the client when you are planning a batch of research.

On previous project we have paid:

  • £30 of Love2Shop vouchers for 1.5 hours of in-person usability testing
  • £50 cash for geographically-specific users to take part in 1 hour of remote mobile usability testing
  • £100 cash for an hour-long discovery research session with people likely to need assisted digital support
  • £100 cash for 45 minute research session with small businesses

Paying incentives through a recruitment agency

If you are using a recruitment agency, ask them to pay the incentive to the participant. They will know how they do this. They may charge a small fee for handling the incentive.

If a participant contacts dxw digital to say they haven’t received their incentive, apologise and contact the agency to let them know.

Paying incentives using shopping vouchers

Where we are paying the incentive, we use shopping vouchers. Normally from Love2shop.

Request the amount of vouchers you need through the #dxw-purchasing Slack channel. They should be in £5 and £10 denominations so they are easier to spend.

In a face-to-face research session, give the vouchers to the participant and have them sign a receipt.

For a remote research session, use the #dxw-purchasing Slack channel to ask for the vouchers to be posted to the participant. Enclose the vouchers in a signed thank you sheet.

Handle the incentive at the start of the research session - either giving the incentive, or confirming that the incentive will be sent. This avoids any suggestion that participants have to ‘earn’ their incentive. It also means you don’t have to deal with the incentive if you want to stop a problematic session.

Other ways to pay incentives

Other options for paying incentives might be a donation to a preferred charity, or a hamper that can be shared with a team.

Withholding incentives

We do not pay the incentive (or any recruitment fee) if the participant doesn’t show up for their session. Although we might still pay the incentive if the participant tries to attend, but is stopped by an emergency or severe travel problem beyond their control.

If a recruitment agency provides a participant who does not match the brief, we will pay the participant their incentive, but we will not pay the agency their recruitment fee.

How we take notes and record research sessions

To capture what happens in our research sessions we take notes and photos, make audio, video and screen recordings, and take copies of paperwork and other material that participants use or refer to.

We use these notes, photos and recordings during analysis to make sure we produce valid findings. And we use extracted quotes, images and clips to illustrate the findings we share.

There is good guidance on Taking notes and recording user research sessions in the Service Manual, and on Capturing User Research in UX Matters.

Here we provide additional, specific guidance for researchers at dxw.

Good practices to follow

  1. Capture only what you need and have consent for

    Good user research practice, and good privacy practice, is to capture only the data you believe you need. For example, don’t make a screen or video recording, ‘just in case’, if you’ll only ever need a transcript of what the participant says.

    Make sure that anything you capture during a session is covered by the consent you’ve collected. If a participant changes their mind about what they consent to, be prepared to not make, or to delete recordings you no longer have consent for.

  2. Do not use your personal devices

    Always use a dxw provided device, rather than your personal device, when taking notes and photos, or making recordings.

    Using our own devices in research means that personal data about research participants can be mixed into our personal data stores, like cloud photo libraries. And also runs the risk of participants, client staff and colleagues seeing our personal messages and images.

  3. Keep your notes and recordings secure

    As soon as possible after the session, transfer any notes and recordings from paper and other devices to your dxw provided laptop. And from your laptop to the dxw Google Drive or to the client’s research store.

    Keep only anonymised notes, quotes and images in tools like Miro and Trello.

  4. Have a fall back

    The one certainty in user research is that things go wrong. So make sure you have other options to capture what you need.

    For example:

    • The participant may decide they don’t want to be recorded in a particular way, so be prepared to switch from video to an audio-only recording, or to just take notes
    • Your preferred device, app or service may not be working properly, so have an alternative like a paper notebook
    • Your note taker, assistant or observer may not be able to join the session, so be prepared to take notes and recordings yourself

    Note that if the participant refuses all recording, including note taking, you may want to abandon the session.

Using your dxw provided laptop

Whenever possible, use your dxw provided laptop for taking notes and making recordings. Our laptops have encrypted drives and we follow good security policies.

With your laptop you can:

  • capture notes to a local file or a Google Sheet
  • record audio using a microphone and the Voice Memos or Quicktime apps
  • record video using an external camera and Quicktime
  • record your laptop screen using Quicktime or Screenshot

If research participants might use your laptop or see your screen during a research session, it’s a good idea to create a separate ‘research’ user account on your laptop with things like notifications and browser autofill turned off.

Using a dxw provided handheld device

Use a dxw provided handheld device, rather than own phone or tablet, to take photos and short videos during interviews, visits and workshops.

Remember to delete the photos and recordings from the device once you have transfered them to your laptop or to Google Drive.

Using the notes template

We have a template for a Google Sheet you can use to take notes during a research session.

You can make a copy of the sheet for each research session, or make a copy for each batch of research and create tabs within the sheet for each research session.

Using sticky notes and paper worksheets

We often use sticky notes and paper worksheets to capture information during research sessions, for example when researching users’ experiences.

Design paper worksheets, and give people advice on writing sticky notes, so you can easily use the information recorded.

Avoid capturing personally identifiable information about participants on sticky notes or paper. If you capture any sensitive informaton on sticky notes or paper, make sure to use a shredder to dispose of it.

Use a dxw provided phone or tablet, rather than your personal device, to capture the contents of sticky notes and paper worksheets.

Using a video conferencing service

Our preferred service for recording remote screen sharing sessions is GoToMeeting. Setup GoToMeeting so that the recorded video is saved onto your dxw provided laptop. And then transfer the video to Google Drive as soon as possible.

Depending on the client, we may need to use their preferred video conferencing service. If the service stores recordings ‘in the cloud’, make sure to download the video as soon as possible and delete it from the service.

Using third-party facilities

If you are using a third party facility like a usability lab, transfer any recordings made by the lab to your dxw provided laptop as soon as possible after the sessions are complete.