Sampling 101 for Creative Testing

What makes a good sample? And how does Swayable ensure unbiased supply?

A good survey experiment depends on the quality of its sample (the respondents). How does Swayable ensure a balanced, unbiased, and targeted group of survey respondents?

I interviewed Swayable’s Director of Product, Anish Bharadwaj, to discuss what goes into sampling at Swayable.

How does Swayable get its survey respondents?

We aggregate responses from a mix of sources, typically served to respondents on their mobile device.

• Reward survey apps: We partner with the best performing survey apps in the market to draw in respondents and compensate them for their attention. We optimize our survey experience on these apps and closely monitor response quality.
• Lifestyle apps and sites: To avoid only targeting users of survey apps, we connect to a range of mobile game and lifestyle apps (and desktop users for some tests) that offer their users in-app rewards for completing a survey.
• Swayable Rewards app - We built out our own rewards app to round out our supply and pull in specific respondents. We have also built a smart router in-house that determines who gets each survey. At any point in time, there is a lot of demand for responses from our clients. The router decides in real time what is the highest value survey that it can serve to each respondent.

How do you ensure the sample is good (e.g. not biased, not made of bots, not people just skipping through the survey)?

That’s a great question that comes up often among savvy clients. We solve for this on two levels.

First, it’s about fishing in different pools. By this I mean our different sources. If we recruited solely from rewards apps, say, our sample would be biased towards downloaders of rewards apps. We ensure a randomized sample by recruiting from a mix of mobile apps across lifestyle, gaming, news, rewards, and a long tail of partners.

Second, we quality check our respondents. We actually have (7) different quality checks that all respondents must pass. These include:

• Attention-check questions - These ensure respondents are legitimate and focused on the survey (e.g. putting a fake brand as an option, and filtering out respondents who claim to be buyers of this brand).
• Quality check filters - These steps remove bots and/or non-legitimate users with (via captcha, IP address matching, deduplications, speed of response, unintelligible responses, locale checking).
• Re-weighting and post-stratification - We remove or re-weight legitimate respondents if the sample gets overrepresented (for example, if a client wants a general population sample, but most of the respondents happen to be women, we will remove the oversampled population to ensure the sample matches the desired audience)

What is a ‘good’ sample size?

This can vary based on the type of survey, population, and level of precision expected. But to give you a useful answer, for our RCT tests, we start seeing statistically significant results to cover our clients’ segmentation needs when the sample size crosses ~200+ respondents per asset being tested.

That said, we typically sample ~1,000 respondents per asset for most of our clients. When clients are in a hurry, or want to reach a specific audience that is hard to recruit, we have brought that number down.

Part of the way we are able to achieve robust results even at the level of very granular segments is through our Machine Learning Inferencing model, which has been fine-tuned through millions of responses over the years.

What kind of audiences can Swayable sample?

All standard demographic and geographic cuts are possible - we can recruit survey samples by location, gender, profession, income, education and more. Often, clients are interested in behavioral cuts, such as people who have purchased a specific brand, or who shop certain stores and categories, and we can recruit those groups as well.

Is there anyone you cannot sample?

As the respondent group gets more niche, they get harder to recruit. For example, recruiting people who have bought perfume in the past year is easier than recruiting people who have bought Tom Ford perfume in the past year. That said, our client success team is exceptional at working with clients and supply partners to acquire a sample of respondents who can provide insight into the performance of their marketing assets.

How long does it take to recruit a sample?

Most of the time we have results within a day, and often within a few hours.

Sometimes, if the sample is very specific (e.g. a client is targeting a hard-to-reach profession), the sample might take a few days to get to completion. But speed is something we consistently receive positive feedback on. And it matters - clients use us to make decisions on marketing campaigns and often want to get in the market ASAP. Particularly our clients in the political space.

What is the best thing about working for Swayable?

Swayable is a group of very smart people challenging each other to build a great business….personally I find that super exciting. Building an early stage startup is hard work but the people bring a lot of passion into what they do, and we have built some fantastic tools as a result!

Anything else people should know?

We’re hiring! Check out our careers page if you want to apply smart thinking to understanding how people think and make decisions.

Takeaways

• A robust, non-biased sample is key to generating accurate insights from a survey

• Swayable avoids bias by recruiting samples from a variety of sources, balancing respondents represent the population clients are looking to understand, and using 7 quality check filters

• Swayable can recruit a sample across most demos in under 24 hours, with powerful targeting capabilities to zero in on specific audiences