Skip to article

Research methods

Compare remote methods by the tasks they suit, how to set them up, and what data they produce.

Research methods

How to choose a method for your research question

At this stage, you already have a list of research questions and hypotheses. If not, read the chapter “Preparing for research.”

Preparing for research

To choose the right method, match each question with one or more methods. This will give you a step-by-step plan and reduce the risk of missing something.

Do not limit yourself to the pattern “one question = one method.” Often, to get the full picture, it is worth checking one question in several ways.

A template connects each topic and its research questions or hypotheses to a business goal.
A template connects each topic and its research questions or hypotheses to a business goal.
MethodHow it worksHelps validate
Unmoderated Scenario-Based Testthe respondent completes a task in a prototype or on a website independently, without a moderator. For example: “Set up the order so it arrives by dinner.”It shows whether users understand the interface, easily find the needed elements, complete the scenario without errors, and perceive the path as convenient.
Preference testthe respondent is shown several variants — screens, copy, or images — and asked to choose the one that seems better. For example: “Which option looks more beneficial?”It helps understand the perception of different solutions, but not usability: which option inspires more trust and looks more attractive, which copy or banners attract more attention.
A/B usability testrespondents are shown different versions of the same screen or scenario — for example, variant A and variant B. Each group completes the same task, and you compare the results: who completed it faster, made fewer mistakes, and where users felt more confident.The method helps determine which interface variant works better — not by taste, but by actual use.
Card sortingthe respondent is shown cards with product elements and asked to group them by meaning in the way they would expect to see them on a website or in an app.It helps design navigation from scratch or check whether the current navigation matches users’ logic.
First Clickthe respondent sees an interface screen and completes a task: find and click the element they would start with.It helps understand whether the design is structured correctly and whether users understand where to click to solve the task.
Five-Second Testthe respondent is shown a screen or page for 5 seconds, sometimes longer if the content is complex, and then answers questions.It helps evaluate the first impression of a page, screen, or product: what people notice immediately, whether they understand the meaning, and how clearly the brand or product message is communicated.

Unmoderated scenario-based testing

The respondent completes a task in a prototype or on a website independently, without a moderator.

Scenario testing helps check: ✔︎ Whether users understand the interface and find the needed elements without hints. ✔︎ Whether they can complete the scenario on their own, in how many steps, and how much time it takes. ✔︎ Where errors, difficulties, or incorrect actions occur. ✔︎ Whether the path feels simple and convenient, and where users most often drop off.

Does not help check:

  • It does not always answer “why” — the reasons for errors are only partly visible through the path or final comments.

What to consider when running it:

  • The test takes 5–10 minutes; the optimal minimum is 15 respondents.
  • The more complex the scenario and the more possible paths it has, the more sense it makes to run a moderated study.
  • It is better to split the prototype into short tasks so the person does not get confused.
  • Add a separate “success screen” to capture the result.
  • The device used for the test should match the prototype.
  • Consider privacy when setting up the test: not everyone is comfortable turning on a camera, and this can also affect completion conversion.
  • The method does not always answer “why” — the reasons for errors are only partly visible through the path or final comments.

Prototypes and test preparation

Preference test

The respondent is shown several variants — screens, copy, or images — and asked to choose the one that seems better. For example: “Which option looks more beneficial?”

The method helps understand the perception of different solutions. It is not about usability, but about impressions: trust, emotional tone, attractiveness. For competitive products, these nuances become part of the strategy because they directly influence the sense of convenience and trust in the brand.

✔︎ Which option inspires more trust and looks more attractive. ✔︎ Which copy or banners attract more attention. ✔︎ Which icons or illustrations feel more appropriate for the product.

Preference-test results showing how 132 responses were split across variants A, B, and C.
Preference-test results showing how 132 responses were split across variants A, B, and C.

What to consider when using the method:

  • Compare only one detail at a time. If the variants differ in too many ways, it will be impossible to understand what influenced the choice.
  • The change should be noticeable, not cosmetic. It is better not to test close color shades, fonts, or button labels — the difference may be too subtle.
  • Phrase the question so that it sets the comparison criterion — for example, “Which option feels more emotional?” or “Which one inspires more trust?”
  • When analyzing results, report the response share for each variant and the confidence interval for the difference.
  • Do not compare your own screens with other services. Differences in logic and style are too large, and the results will be incorrect.
  • Remember that Preference tests and A/B tests provide different information. The first reflects subjective perception; the second reflects real behavior. The results may not match.

What results you will get:

  • The percentage of people who selected each screen.
  • Detailed respondent comments explaining why they chose one option or another.

How to identify the winning variant

Before running a comparison, choose the primary metric and define what difference would change your decision. After the test, report the observed difference between variants and its confidence interval. Report differences in proportions, such as first-click success rates, in percentage points. Report differences in means in the metric’s own units, such as seconds. If the interval includes both a meaningful improvement and no effect, the data are still inconclusive. Whether the finding generalizes to the wider audience depends on who was recruited and how.

How to interpret the comparison

A/B usability test

Respondents are shown different versions of the same screen or scenario — for example, variant A and variant B. Each group completes the same task, and you compare the results: who completed it faster, made fewer mistakes, and where users felt more confident.

The method helps determine which interface variant works better — not by taste, but by actual use.

For example:

✔︎ Which page version helps users complete the scenario faster. ✔︎ Which button layout reduces errors. ✔︎ Which option generates more conversions.

How it differs from a preference test:

In an A/B test, people perform a task and you measure effectiveness. In a preference test, users simply choose which option they like visually or which one seems clearer.

What to consider when using the method

For reliable results, it is better to recruit enough respondents in each group. We explain how to calculate the required number of respondents in the chapter “How many respondents you need for quantitative research.”

Sample size for quantitative research

Pathway offers a choice between creating a standalone test and creating a split group to compare variants.
Pathway offers a choice between creating a standalone test and creating a split group to compare variants.

Card sorting

The respondent is shown cards with product elements and asked to group them by meaning in the way they would expect to see them on a website or in an app.

  • In closed sorting, the groups are predefined.
  • In open sorting, respondents create the groups and their names themselves.

The method helps not only design a structure from scratch, but also check how well the existing navigation matches the way users group information:

✔︎ How well do product navigation and structure match users’ logic?

✔︎ Which categories or sections should be merged or renamed?

✔︎ What is the best way to organize catalog and menu navigation?

✔︎ Which elements are difficult to place?

What to consider when using the method:

  • Sorting does not give you a ready final menu — you need to analyze the grouping logic, find patterns, and only then build the structure.
  • Number of cards: 30–50 is optimal. If there are too many cards, respondents get tired.
  • Run it with several people: one result will not tell you anything, but with 20–30 participants stable patterns will appear.

What results you will get:

  • For open sorting: catalog categories or menu items identified by respondents.
  • For closed sorting: the distribution of elements across groups made by respondents.
A card-sorting report shows how participants assigned cards to categories.
A card-sorting report shows how participants assigned cards to categories.

First Click

The respondent sees an interface screen and completes a task: find and click the element they would start with. The method shows whether the design is structured correctly and whether users understand where to click to solve their task, for example: ✔︎ How easy it is to find an element on the page. ✔︎ How respondents use navigation, for example to understand which menu item people look in for the needed action. ✔︎ How effective the menu is, for example whether the respondent can find a specific product in it.

What to consider when using the method:

  • Do not use button names or menu item labels in questions. Where would you click to pay? Imagine you want to buy this product. What would you do?
  • When comparing screens, estimate the difference in your chosen metric and its confidence interval.

What to consider before drawing conclusions

  • Add comments to clicks. Ask “Why did you click exactly here?” — this helps understand motivation and hidden logic that is not visible on the click map.
  • Evaluate choice speed. It matters not only where people clicked, but also how quickly. Long hesitation may indicate doubts or an interface problem.

What results you will get:

  • The main metric: the percentage of users who made the correct first click.
  • Additionally, you can look at where people clicked incorrectly and which elements competed for attention.

Five seconds

The respondent is shown a screen or page for 5 seconds, sometimes longer if the content is complex, and then answers questions. The method helps understand:

✔︎ What first impression the page creates. ✔︎ Which elements users notice immediately. ✔︎ How they understand the meaning and purpose of the page or product. ✔︎ How clearly the message is communicated and whether it matches brand expectations.

An example of the instruction shown before the image in a Pathway five-second test.
An example of the instruction shown before the image in a Pathway five-second test.

What to consider when using the method:

  • Do not hint at what exactly people should notice or remember.
  • Do not tell respondents in advance that the screen will be shown for only 5 seconds — simply limit the display time.
  • After the display, ask 1–3 open questions, for example: What do you remember from this page? What do you think this product, website, or screen is? What did you like or, on the contrary, find confusing?
  • If the interface contains a lot of text or many elements, increase the time to 10–20 seconds to keep a balance between first impression and understanding.
  • To compare different design versions, for example two landing pages, a Preference test is a better fit.

Preference test

What results you will get:

  • Qualitative answers: what people remembered, how they described the page in their own words, and what emotions they felt.
  • Quantitative data: the share of participants who correctly identified the page goal, noticed key elements, or mentioned the brand.

Other methods that help measure opinion, attitude, satisfaction, and experience

Above, we covered the main unmoderated research methods. In many of them, after scenarios or tasks, you can add multiple-choice questions, satisfaction scales, or open questions to understand what people felt and thought.

Before the study begins, you can also learn about the respondent’s experience to interpret the results correctly: whether the person has used the product, faced similar tasks, or belongs to the target audience.

The same question types can also be used in separate surveys when you are not testing an interface but want to collect users’ opinions or experience.

Scale

The respondent evaluates an object or experience on a scale — numeric, emoji-based, or star-based. The method helps measure subjective perception: satisfaction, importance, or ease of use, for example:

✔︎ How easy was it to complete the task? ✔︎ How do users rate the importance of factors, such as delivery speed? ✔︎ Satisfaction level with the product or a specific feature. ✔︎ Impressions of the interface and emotions while using it.

A participant rates how easy it was to place an order on a one-to-five-star scale.
A participant rates how easy it was to place an order on a one-to-five-star scale.
A scale report showing the distribution and mean rating for the importance of preserving filters.
A scale report showing the distribution and mean rating for the importance of preserving filters.

What to consider when using the method:

  • The optimal scale length is 5 or 7 points.
  • Many scales in a row tire respondents.
  • Labels are required — without them, people interpret numbers differently.
  • A scale shows subjective opinion, not actual behavior.
  • For serious research, such as satisfaction, importance of factors, CSI, or NPS, it is better to use numeric scales. Emoji can be left for quick reactions or “light surveys,” where engagement is more important than precise data.

What results you will get:

  • Average score
  • Overall statistics

Matrix

The respondent evaluates several factors at once on a scale, for example from “not important at all” to “very important” or from “poor” to “excellent.”

The method helps compare several parameters at the same time and see what matters most to users. For example:

✔︎ Which product characteristics influence choice the most? ✔︎ How well does the product perform different tasks, such as speed, simplicity, and reliability? ✔︎ How do users rate different aspects of the experience: navigation, visual style, loading speed?

A matrix report showing ratings from 1 to 5 for each type of personal data.
A matrix report showing ratings from 1 to 5 for each type of personal data.
A participant uses a matrix to rate the importance of each online banking feature.
A participant uses a matrix to rate the importance of each online banking feature.
A matrix lets the participant match shopping situations to the payment methods they use.
A matrix lets the participant match shopping situations to the payment methods they use.

What to consider when using the method:

  • Do not overload the question: 5–10 factors is optimal.
  • Make sure factor wording is clear and does not overlap.
  • Use the same scale for comparison.
  • The method does not explain “why” — for that, you need interviews.
  • Compare the means directly. Report the difference in scale units and the confidence interval for that difference.

What results you will get:

  • Factor ranking

Open question

An open question is one of the simplest and at the same time most difficult ways to get information. It helps:

  • In surveys, when the topic is new and you do not yet know how to classify possible answers.
  • In tests, to learn after a scenario how a person explains their actions, impressions, or difficulties.
An open-ended question asks the participant to describe difficulties in their own words.
An open-ended question asks the participant to describe difficulties in their own words.

The difficulty is that respondents quickly get tired of writing detailed answers. Use open questions selectively — only where it is truly important to hear the user’s “live voice.”

What to consider when using the method

  • Do not overload respondents: 1–2 open questions per study are enough.
  • Place the open question at the end of the block, after the main choice or rating.
  • Ask for an example: this adds specificity and helps interpretation.

What results you will get:

  • Text answers from which you can identify themes, patterns, and insights.
An example of an AI-generated summary for an open-ended question in Pathway.
An example of an AI-generated summary for an open-ended question in Pathway.
The summary groups answers by theme and shows how many participants mentioned each theme.
The summary groups answers by theme and shows how many participants mentioned each theme.

Multiple choice

The respondent is given a question with several answer options. They can choose several that apply to them. The method helps understand which options are truly meaningful and what the choice is based on, for example:

✔︎ Which payment or delivery methods are used most often. ✔︎ Which communication channels are considered convenient: email, messengers, calls. ✔︎ Which barriers and difficulties occur more often than others. ✔︎ Which information sources help people decide about a product or service.

A multiple-choice question asks which delivery methods the participant uses.
A multiple-choice question asks which delivery methods the participant uses.

What to consider when using the method:

  • Options should be clear and not overlap with each other.
  • Do not use multiple choice without a qualitative base: first collect options through interviews, desk research, or open questions, and only then validate them in a closed list.
  • Avoid overly long lists: 6–10 items is optimal. If there are more than 12–15 options, group them or use filters, for example first category selection, then clarification.
  • For sensitive topics, such as income or health, phrase questions carefully and add a “Prefer not to answer” option.
  • Use an “Other” option so the respondent does not get stuck if they cannot find a suitable answer.

What results you will get:

  • Quantitative distribution across options.
  • Comments in the “Other” option.

Additional settings: when and why to use them

These settings help make surveys and tests more comfortable for respondents and more accurate for you. Below is a short overview of how and when to use each setting.

Optional question. The respondent can skip it if they do not have experience or are not ready to answer.

  • Suitable for sensitive topics: income, health, family status.
  • When a person may not know what the question refers to — for example, “night delivery.”
  • If there are too many optional questions, the answers will be incomplete.

Allow selecting multiple options. Use this when the respondent may have several correct answers. This makes the data fuller and closer to real experience.

Shuffle answer order. If the list has logic, for example “18–24,” “25–34,” keep the order. If there is no logic, enable shuffling. This helps:

  • Reduce position bias: frequent selection of the first items
  • Prevent mechanical answers
  • Improve data accuracy

What results you can generally get from unmoderated research

Which indicators you need depends on the task and the research question.

Success rate

Define task success and an acceptable outcome for the scenario before testing. Report how many participants succeeded, how many attempted the task, and where they struggled.

Interpreting differences between variants

Efficiency

How much time a person needs to complete the task. In scenario tests, a long time points to difficulties completing scenarios.

Summary cards show completions, abandoned attempts, and mean and median completion times.
Summary cards show completions, abandoned attempts, and mean and median completion times.

Problems

If users did not complete the task or spent a lot of time, you need to analyze the reasons — the problems they encountered. Path analysis and video review will help.

A path diagram shows how participants moved between prototype screens.
A path diagram shows how participants moved between prototype screens.

Feature memorability

Within one test, you can check how well users remember how to use a feature. First ask them to complete the scenario of turning the feature on, and after some time — the scenario of turning it off, to compare how quickly and confidently participants handle the task.

Satisfaction

A report shows the distribution and mean of task-ease ratings on a five-point scale.
A report shows the distribution and mean of task-ease ratings on a five-point scale.

Experience, opinion, and attitude

With multiple-choice questions, scales, matrices, and open fields, you can learn how people perceive the product, evaluate the experience, and relate to different solutions.

Checklist for checking tasks and questions in unmoderated research

Use this checklist to quickly make sure tasks are clear to respondents. Below you can read more about each point and see examples.

Add context before the task. Without a couple of lines about the situation, a person will not understand what to look for and why. Their behavior will be random.

A Context block introduces the participant to the situation before the tasks begin.
A Context block introduces the participant to the situation before the tasks begin.

One idea in one question. Do not combine two questions into one — the respondent will answer only part of it or get confused.

Keep wording short. Long and overloaded phrases make perception harder and make respondents lose their place.

Use real scenarios. Tasks should sound like familiar human actions, not like interface descriptions.

Avoid unrealistic roles. Do not place people in situations far from their experience — this reduces the reliability of results.

Minimize open questions. In unmoderated tests, open answers provide less data and are harder to analyze. It is better to use scales or answer options.

Do not name interface elements. If you mention a button or link by name, the person will start looking for exactly that element rather than solving the task.

Check neutral wording. Questions should not contain evaluative words.

Scroll to top