Skip to article

Audience and sample

Define audience segments, recruit participants, and choose a sample size for the study.

Audience and sample

Who to recruit and how many participants you need

To get results you can work with later, you need to understand whose answers will be useful.

It is important to consider respondents’ experience and behavioral factors. If you are building a competitor to “1C,” you will need accountants; if you are building a ticket-purchase aggregator, you need to recruit people who travel often.

Demographic factors such as age, geography, gender, and so on can also matter if you know that users from different demographic groups behave differently.

Most often, you can discuss the audience with the product manager.

Below, we will look in detail at how to focus on the right segment and invite its representatives to take part in the study.

How to define audience segments for research

A user’s history and context affect their experience with our product. For example, if you are building a service for delivering medicine from pharmacies, the needs and habits of older people will differ strongly from how younger people behave when buying medication. Product decisions, marketing campaigns, and other activities should be aligned with the different experiences of your customers, so it is important to conduct research with relevant representatives of your user groups.

When defining the target audience for a study, you need to answer two important questions:

Which criteria describe the group whose answers will be relevant? When defining criteria, consider: demographics such as age, gender, and region; income level; experience using our product, such as active users, newcomers, and others; experience using competitors. If you are planning a general study of audience needs or pains, or if the opinion of your typical user is enough, you can use broad criteria for your target audience, such as region and age, plus basic activity, or choose the most valuable and promising segment that brings more profit.

To decide, think about:

  • How fragmented your target audience is: whether you focus only on men aged 35–50 with average income, or whether you also target younger and older customers.
  • How different your direct competitors are: the different experience of their users will affect how they see your product, and it may be important to learn different opinions, for example WB and Ozon customers and their perception of Yandex Market.
  • Whether the results will be used only to attract new customers, retain existing ones, or both. If both, you will need to compare the perception of these groups.

You can read about how many respondents you need when working with several groups in the chapter “How many respondents you need”

How many respondents you need

What the size of the user pool can tell you

During sample selection, you will most likely be able to estimate the size of the user pool that matches the criteria for the segment you need. Sometimes it can be very small, for example 20–30 people, and sometimes, on the contrary, very large, for example around 100,000 people.

If the pool is very small, consider relaxing the search criteria to recruit a relevant number of respondents. Beyond its importance for recruiting, this signal may mean that the overall population of users with the target criteria is so small that it may not make sense to treat them as significant representatives of your target audience; you may need to reconsider your focus. Also, the smaller the pool, the longer recruitment will take.

If the pool is very large, on the contrary, consider whether it would be useful to specify the criteria more precisely, focus on a particular user group, or study differences inside it so that you do not end up with conclusions that are too general and blurred. However, when building a sample from your “own” users, a large pool also sends a positive signal: the size of the group means it is important within your target audience, so its opinion matters for product development.

Recruit respondents “from inside” or “from outside”?

When planning the research design, you need to decide whether you can get the needed answers from your users or whether you need to find an audience that is not familiar with your product.

This is easy to determine by asking yourself key questions related to the business problem of the study:

  • What knowledge do we want to gain? If the goal is to understand perception and work patterns specifically with our product, invite existing users. If you need to learn about needs and pains in the broader domain, it is worth adding an external audience to the sample.
  • Which metric do we want to influence using the research results? If we want to increase activity among current users, it is important to learn about their experience, for example if we aim to reduce churn rate. If we want to attract new customers, it may be important to learn about the needs of an external audience.
  • Are we studying an update to existing functionality, or planning to create something new? In the first case, it is important to check how already acquired customers will react to the new interface or experience. When something new is being designed, it may be important to hear how this already works in the industry for users of other services and solutions.
  • Do you have clear competitors, and does experience using them matter? The presence of other strong players in the market means that users may already have established patterns and habits based on best practices from other products. The influence of this experience is important to consider so you avoid mistakes and build a competitive product; to do that, you need to talk to customers of other services.
  • Does the product have enough active internal users to answer the research questions? If your service has only recently started developing, the active customer base may be too small for relevant results. In that case, an external audience or a mixed sample, with some internal and some external users, can help.

Where to find external respondents

To invite users who are not familiar with your product to a study, you can use the following options:

  • Recruiting agency services. Companies that specialize in selecting respondents for research in specific countries and regions can recruit respondents for you end to end. This option is convenient and frees up your time, but it is significantly more expensive than the others.
  • Recruiting respondents through the unauthorized area of your service. If you have tools for setting up pop-up surveys on product landing pages that unauthorized users can reach, you can invite interested people to take part in your study. The downside is low conversion and sample bias: participants will already be somewhat interested in your product because they reached its landing page.
  • Manual respondent search in thematic communities or other places. This is the most time-consuming and labor-intensive option. However, it can help when you need representatives of a very narrow segment that is hard to reach in other ways, for example owners of private dairy farms in southern Russia.
  • Respondent panels and the capabilities of survey and unmoderated-testing platforms. These services often let you distribute a research link to your own audience and recruit external respondents through their panel. This option is often optimal because it helps achieve high conversion quickly at a reasonable price.

For example, Pathway offers more than 120 parameters for finding respondents inside the platform. This helps you find research audiences from Russia, Kazakhstan, Uzbekistan, Armenia, Georgia, and other CIS countries in just a few clicks. On average, Pathway customers save around 800,000 rubles per year on respondent recruitment.

External recruiting settings in Pathway include group size, age, region, and additional filters.
External recruiting settings in Pathway include group size, age, region, and additional filters.

VkusVill UX research case study

B2C and B2B audience

Finding the right audience for a study can be difficult, and it becomes even more complicated when researching a B2B audience.

Let’s recall how these two audiences differ:

B2B, or Business-to-Business, is a business model in which a company sells its product or offers services to other organizations for work-related purposes. B2C, or Business-to-Consumer, offers products and services to individuals, consumers who use the service for personal needs.

The overall populations of user segments in B2B are usually significantly smaller than in B2C.

One product can have both a B2B and a B2C audience at the same time. Using the same online menu for cafes as an example, the business needs to make sure the product is interesting and clear for cafe owners (B2B) and convenient for the end customer (B2C). Such products are often called B2B2C: business-to-business-to-consumer.

Despite the difficulties, a B2B audience is reachable. Julian Della Mattia, founder of the From Finders to Builders podcast, and Ksenia Zhebrovskaya, Brand and Business Development Manager at Pathway, share techniques that make B2B respondent recruitment easier:

  • For internal recruitment, collaborate with support, customer success, sales, or client curator teams. They build and maintain relationships with customers, so they can definitely help you attract participants.
  • Attend industry events. Join online communities, go to meetups and conferences related to your field, where you can meet your target audience. Be human, open, and transparent: talk to people, explain in detail what you plan to do and how you would like to involve them.
  • If recruiting is very difficult, think about what you can change in your approach. This may be the participant selection criteria, the reward you offer, or the invitation text itself. Treat all of these as hypotheses and test them one by one. Sometimes changing the wording of the invitation is enough to completely change the situation.
  • Be creative. Ksenia gives an example from her own experience. When the team needed to study the Hong Kong market with a Chinese-speaking audience, they hired a talented marketing manager from the startup community. She used her contacts and brought exactly the sample they needed for in-person focus groups in their office. This case shows perfectly how personal connections and nonstandard thinking can lead to successful recruiting.
  • Identify the recruiting panels most relevant to your target audience. For example, Pathway has access to one of the largest respondent panels in Russia and the CIS countries. If you are interested in an international market, Pathway’s cooperation with Tandems, TestingTime, and PolarInsight may help. Tandems are valued for recruiting respondents from Brazil, TestingTime works well with participants from the Netherlands, and for studies focused on the UK, good options include FieldworkHub or People4Research.
  • Broaden the search focus. Sometimes suitable B2B respondents can be found among a wider audience, even B2C. For example, if your product helps teams manage processes and build a transparent corporate culture, many B2C users may have experience with similar tools at work. In that case, everything comes down to effective screening or even automatic targeting on platforms like Prolific or Respondent.io.

How many respondents you need

Unfortunately, there is no simple answer, so first we will explain what it depends on.

Qualitative research helps uncover problems and generate hypotheses. Quantitative research helps estimate metrics and test hypotheses.

The required number of respondents will differ for each type. We explain how to calculate it below.

For qualitative research

For an initial qualitative usability round with one fairly similar user group, around five participants can be a starting point. If different scenarios or audiences matter, recruit for each of them. In unmoderated testing, allow for incomplete or unusable sessions.

Why this number?

NN/g’s model for small usability testing rounds

The NN/g model illustrates why each additional participant tends to uncover fewer new issues within a qualitative testing round.
The NN/g model illustrates why each additional participant tends to uncover fewer new issues within a qualitative testing round.

When selecting a sample, consider whether sessions are moderated, how many responses may be unusable, and which groups you need to compare.

For quantitative research

Quantitative research helps estimate a metric to a chosen precision or compare variants.

With a significance level of 5% chosen in advance, a result with p < 0.05 is called statistically significant. Under the test’s assumptions, the observed data are hard to reconcile with the null hypothesis. Assess effect size, practical importance, and generalizability separately.

Population — all people who belong to your product audience. For example, everyone who shops online, or everyone registered as self-employed.

Sample size is the number of observations available for analysis. Generalizability depends on how the sample was selected and on data quality.

The margin of error describes precision under the calculation’s assumptions. Assess potential recruitment bias separately.

How many respondents you need for quantitative research

If you need to estimate a proportion precisely or compare variants, calculate the sample size for that objective. For an estimate, specify the desired precision. For a comparison, specify the smallest meaningful difference, expected baseline, significance level, and power.

Around 100 independent observations give a rough margin of ±10 percentage points for one proportion at the 95% confidence level in the least precise case. Comparing two variants requires a separate calculation.

The table below gives planning estimates for a single proportion.

How to calculate the number of respondents for quantitative research

For estimating one proportion, sample size depends on the acceptable margin of error, confidence level, and expected proportion.

If the expected proportion is unknown, use 50% for a conservative calculation.

When comparing variants, the smallest meaningful difference is a change that would affect the team’s decision.

The table shows approximate margins of error, in percentage points, for one proportion.

Sample sizeMargin of error
3018%
4315%
9610%
3845%

These planning estimates assume a large population, independent random sampling, an expected proportion of 50%, and a 95% confidence level.

For planning, specify the margin of error and confidence level. In an interval calculation, 95% refers to the confidence level. For a hypothesis test, specify its significance level α separately, for example 5%.

The calculator has controls for the maximum margin of error and a percentage level.
The calculator has controls for the maximum margin of error and a percentage level.

Respondent rewards

The question of whether testing participants should be rewarded often comes up in product-community conversations. The answer is clear: yes, and here is why.

  • Higher participation conversion. A reward motivates respondents to participate in research in exchange for material compensation.
  • Sample normalization. Active respondents who are willing to share opinions for free are most often at the extremes of the sample: very satisfied, very upset, or with a lot of free time. Feedback from such users cannot reliably reflect the situation in the population, which includes busy or quiet users with neutral experiences. Financial motivation helps attract them too.
  • It formalizes the relationship between researcher and respondent. Participation in research is a kind of work that requires concentration and involvement in the process. Knowing that the participant will receive a reward makes them a more responsible respondent and helps the researcher feel entitled to ask many questions and follow-ups.
  • It communicates respect for the user. Paying for spent time and useful information is good practice that helps strengthen the company’s reputation and can even increase customer loyalty.

What if we cannot pay for participation?

In young products or under budget constraints, it may be impossible to provide rewards for research participation. You can still talk to respondents in this case: research in imperfect conditions is better than no research at all.

To minimize sample distortion in this situation:

  • Try to recruit participants proactively, inviting them through online mailings or direct contact, so you reach interested respondents who are not necessarily actively seeking communication with you themselves.
  • Control key demographic parameters and product-activity data using analytics or qualification questions. This will help make sure you are getting a balanced sample.
  • If possible, expand the candidate pool, for example from 1,000 to 5,000–6,000 people, to compensate for the drop in conversion without rewards.

What to give for participating in a study

However, if you plan to reward respondents yourself or distribute a test created in Pathway to your internal audience, you can consider these options:

  • Certificates for popular regional marketplaces. A universal reward that almost every participant can use. It works best for longer interviews or moderated tests.
  • Merch. Branded gifts are valuable because they are unique and create a sense of belonging to your product. If the merch is also useful, it becomes an excellent reward.
  • A promo code for your service. If your service provides widely used services, such as buying electronics or booking hotels, a large discount or the ability to get something for free is a good incentive. This option works well for quantitative surveys and unmoderated tests because it is easy to implement online.
  • For the premium segment: experience gifts and compliments. High-income audiences are difficult to motivate with money, and your study will most likely attract the most curious and open among them. Such respondents can be interested in unusual, status-oriented gifts or compliments, for example sending a fruit basket or assigning a special status in your product.

To calculate the reward amount for moderated and other sessions, you can use one of the online calculators that account for region, recruitment difficulty, and participation effort. For example, Ethnio.

Ethnio’s incentives calculator

Scroll to top