Sample Specification: The Most Important Research Design Decision
When designing a research study, there are many critical decisions to be made – Qualitative or quantitative? Online or in-person? Tactical or strategic in focus? While these questions are undoubtedly important, the most important decision is sample design. Even the most well-designed and analyzed study can be misleading if you talk to the wrong folks.
Most sample design has at least four interrelated elements:
- Sample Size
- Decision-making role (primary, influencer, etc.)
- Category participation
Specifying criteria in each of these areas may seem fairly straightforward. Yet it’s important to check your assumptions before defaulting to ‘standard spec’s.’ We see many RFP’s that use standard language like this:
N=500 Men and women 21-55 years old in the U.S. who have sole or shared decision-making authority for [CATEGORY NAME] and have made one or more purchases in the past 3 months.
This innocuous description carries many implied assumptions that should be challenged before finalizing the sample design:
Assumption #1: Light and ‘lapsed’ category users are unimportant to growth.
It’s easy to see how light users are an important source of volume for new brands and small brands, who are not yet established. What may be more surprising is how important they are to market share leaders. According to analyses by Bryron Sharp, light users account for a larger portion of volume for leading brands. So-called ‘lapsed’ category users may actually be loyal brand users, who simply have very long purchase cycles. Failure to include these respondents risks missing important insights.
Beyond these strategic reasons to include lighter or lapsed users, there are practical ones. The more narrow the sample requirements, the higher the cost of a pool of sample respondents. Often, the lower cost per respondent associated with a higher incidence sample can result in a larger overall sample – and the same number of category users – for the same cost.
In a recent study of frozen turkey users, we encouraged our client to include anyone who had purchased one or more frozen turkeys in the past year. The increased incidence allowed us to go from a sample of n=1000 to a sample of n=1500 for no additional cost. We learned that light turkey users are no more likely to purchase the leading brand, Butterball, than more frequent users, which was contrary to expectations. Including lighter users, many of whom proved to be ardent turkey lovers, also helped us identify new marketing opportunities by helping us understand what other categories such as fresh whole turkey and turkey parts, were satisfying their turkey cravings.
Assumption #2: New category users are unimportant to growth
Few brands of any size can afford to assume their category will grow without reinforcing category level benefits. Category leaders in particular have a special interest in promoting category penetration. New users are often more open to trying new brands or innovative offerings. By including a concept evaluation exercise (e.g., purchase interest, heat map, etc.), a survey can determine which non-category participants represent the most likely target for growth.
Recently, we helped the Healthy Pet Company size the potential market for natural cat litter by including clay-only litter users as well as occasional and exclusive natural cat litter users in our research sample. Clay online litter users’ response to the new product concept revealed strong interest among a segment of clay-only users who were looking for better odor control. This finding shaped how Healthy pet went to market with its wood-based natural litter brand, okocat, in everything from package messages to shelf placement.
Assumption #3: There’s little volume among young Millennials and older Boomers
It’s a myth that younger consumers and older consumers are of less value as a marketing target. Many have significant incomes, families and ongoing needs across a range of categories.
In our turkey study cited above, we learned that consumers 55 years and older are in fact among the heaviest purchasers of frozen turkey! According to Mintel, adults 18 and over spend an average of $106 per week on groceries. Millennials 18-34 spend more - $114 – and 55-65 year olds spend only a little less ($98). Even those over 65 still spend $88 a week on groceries on average.
In electronics, online shopping and internet services, it is especially dangerous to assume that older consumers are not participating at a rate equal to or close to that of younger consumers.
Assumption #4: All recent users are equally relevant
In most categories, not all respondents are equally interesting. Heavier users, brand advocates, new brand users, concept acceptors or even lapsed users can reveal important insights, assuming there are enough of them in the sample to analyze. For this reason we often recommend larger sample sizes that enable us to ‘slice and dice’ without fear of unstable sample base sizes.
Another reason to ensure adequate representation of these ‘super respondents’ is the ability to re-contact them for follow up studies. Many surveys or communities can be well-served by a follow up study – interviews, online focus groups or even a pop-up community or ‘village’ can be an effective way to efficiently leverage the cost of recruiting over more than one study.
Getting to the ‘Right’ Folks
Challenging the status quo in sample design can be the difference between study that yields insights or just a lot of data. Here are three general guidelines for making sure you get it right.
Don’t Skimp! In most research, sample costs represent a small fraction of the total cost, so it is wise to gather data from as many people as you can afford or think you will need. It’s far better to err on the side of too much sample than too little. Even when contracting with a commercial panel, sample is cheap relative to the overall cost and effort of a research study. When using internal panels, the cost can be minimal.
When in Doubt, Go Broad! As the cat litter and turkey examples above demonstrate, volume may be sourced from unlikely places. There’s little risk of going broad as often a larger, lightly screened sample costs the same as a smaller more precise one and it’s easy to filter out respondents who turn out to be unimportant. Unless you talk to everyone who may be relevant, you may never know what you missed!
Follow up! In any study, there are some respondents who are more intriguing than others. Many studies could benefit from a follow up study among a subsample of the original respondents. To that end, be sure to work with panel providers that allow respondents to be recontacted.
 How Brands Grow by Byron Sharp