In a partnership with the digital media strategists at Harmony Labs, Swayable tested to see if engagement on social media was related to content persuasiveness.
The Test:
Eighteen video ads were created covering a range of progressive issues from immigration to tax reform. All videos were promoted on Facebook and engagement metrics for clicks, likes, shares, and time spent were collected. Swayable used random control trial experiments to measure persuasiveness for each of the videos through Swayable’s platform.
Participants were divided into control and test groups assigned to watch either one video ad or an irrelevant PSA video for the control group (the placebo). All respondents were then surveyed on whether they agreed with the video and if they’d take action. This method allows Swayable to measure opinion change as the difference in agreement between the control group and the test groups given video ads--exactly the way a new drug would be tested for effectiveness.
The Results:
The experiment surveyed 10,000 people in 24 hours and found no correlation between the opinion change and engagement metrics. If the two were correlated, these graphs would show a line. The haphazard dots prove there’s no correlation at all.
The Conclusion:
Engagement metrics aren’t a viable proxy for whether content had its intended impact.
For the full experiment write-up see Wired’s coverage.