Reports · Methodology
How the reports are made
Randomized, placebo-controlled message tests, each video read against its own wave's placebo
Every respondent sees one video or the placebo
A respondent answers a few questions about themselves, is assigned at random to watch one of the test's videos or a placebo video, then answers the ballot question and one question that checks what they understood. Because the assignment is random, the difference between a video's viewers and the placebo's viewers is the video's effect alone.
Each wave has its own placebo, and every lift in a wave is against that placebo. Waves are separate samples; a lift in one wave is never read against another wave's placebo.
A lift is the video's level minus the placebo's, in percentage points
Ballot support
Comprehension
A lift marked ≈ is derived from rounded published percentages. The computed waves carry a margin of 1.645 standard errors (a 90% interval); the source reports for Waves 1 and 2 supply no interval endpoints, so none are shown for them.
Detectable means the lift met the wave's stated criterion against its placebo
Detectable
Inconclusive
The criterion is a statistical decision rule. It does not mean there is a 90% chance any particular person changed their mind.
Every level is weighted to the wave's population
Waves 1 and 2 were weighted by the source to age, gender, race, education, party and 2024 presidential vote.
A computed wave is raked by iterative proportional fitting to targets for Massachusetts general adults (18+) on education, party and 2024 presidential vote, drawn from the audience store's sources. Weights are normalized to a mean of 1 and capped at 5.
The weighted total is not an effective sample size.
A subgroup's bars are levels, not lifts
Each bar is the share choosing Yes inside one subgroup and one arm. A mark means that video differed from the placebo inside that subgroup at the wave's criterion; it does not compare videos with each other.
A hatched bar is a small sample: the source's own flag for Waves 1 and 2, fewer than 50 actual respondents for a computed wave. No multiplicity adjustment is applied across subgroups, so with dozens of subgroup comparisons some marks are likely chance.
Where each report's numbers come from
Waves 1 and 2 are transcribed from the source's reports. Wave 3 and the Open Primaries Message Test are computed from their tests' own responses, and both came from seeded demo runs: their respondents are simulated, so their figures exercise the pipeline and are not measurements.