When it comes to consumer insights to drive marketing strategy, the Internet has made testing and surveying easier than ever. What started with focus groups and phone surveys has moved online, but there are several different methodologies for brands to pick from.
When you need to gain trustworthy and actionable consumer insights, what methodology is best? Below we’ll highlight three different methodologies and how they stack up.
Although randomized controlled trials are most commonly associated with drug testing, the methodology has also been successfully deployed in consumer insights and is the gold standard for determining causation. Respondents are randomly segmented into two groups, a control group and a treatment group. The control group is shown a completely unrelated message or visual at the beginning of a survey, while the test group is shown the creative content that is being tested. Both groups answer the same questions after viewing their assigned content. Responses from the control group are used to create a baseline. Comparing results from the two groups will show the true impact of the tested content. The benefit of a randomized controlled trial is reduction of biases and being able to compare to a baseline, thus having evidence confirming that the asset alone caused the change in sentiment that was witnessed in the survey responses.
A/B testing involves head-to-head testing of two (or more) different versions of content with the metric of success being click-throughs or other platform-specific engagement metrics, such as “likes” on Facebook or “views” on YouTube. Respondents are randomly assigned one content asset. The engagement with each asset is compared. For example, a company could do an A/B test by running two ads for a short period of time, and decide to keep the one that has the greater engagement. A/B tests can help optimize campaign content at the bottom of the marketing funnel where the desired outcomes – clicks, interactions, or purchases, to name a few – occur directly on the ad unit itself. There is no baseline in A/B testing to compare the effect of the content to, so the results of the study are all relative between the pieces of content. A/B testing is most useful for direct response campaigns where a click-through or conversion is the goal of the content. It is less useful when the campaign goal occurs outside of the user flow of the ad unit, such as a change in attitude or intent, or where a purchase will take place in the real world.
With Pre/Post Testing, each respondent is asked a set of questions before and after an experimental treatment in order to estimate how much the treatment affected them. The post test, which might be administered immediately or days later, is compared to the pre test to measure this change. This type of testing is easy to execute with smaller sample sizes, and it gives an estimate of changes in attitudes and opinion at the individual level. However, as a methodology, it introduces biases that cannot be fixed. When the same person is asked the same question repeatedly, with a content treatment in between, it is obvious to them that the intent is to produce a change in their answers, so this kind of introspective, self-reported change in opinion is not typically regarded as strong evidence in social science research.
Randomized controlled trial testing is the gold standard for measuring persuasion. It gives decision-makers direct knowledge of cause and effect, going beyond the kinds of simple interactions available to A/B testing. Measuring the impact of a message requires knowing the opinion or intent of a consumer, which A/B clicks cannot meaningfully measure. People often click or interact with content (or don’t) for reasons that have nothing to do with opinion-change, or change in purchase intent. Additionally, RCT allows for comparison against a baseline or control group and does not rely on individuals to self-report changes in their opinion, providing more reliable and robust data than Pre/Post. In the words of Swayable CTO Valerie Coffman, PhD, “RCT is the required practice in drug trials, and any context when knowing what works is mission-critical. Using anything less in a business context means getting decisions wrong.”