Massachusetts Message Testing · Wave 3
Massachusetts Wave 3: What the test found
October 29, 2026 · 2,464 simulated respondents (2,464 weighted) · six treatment videos plus placebo
Four of six videos showed detectable ballot support differences; one showed detectable comprehension differences
- Population
- Respondents who completed the test, weighted to Massachusetts general adults (18+)
- Fielded
- October 29, 2026
- Sample
- 2,464 respondents · 2,464 weighted
- Control
- Placebo video · 356 weighted
- Threshold
- 90% criterion (two-sided two-proportion test)
Ballot support
The weighted percentage selecting Yes, compared with the placebo arm.
Comprehension
The weighted percentage choosing the correct description, compared with the placebo arm.
The 90% criterion (two-sided two-proportion test) is a statistical decision rule. It does not mean there is a 90% chance any particular person changed their mind.
The six videos and their results
35 s
01 · Danielle Allen
352 actual respondents
Ballot support62.8%
vs 48.6% placebo · +14.2 ppDetectable
Comprehension50.1%
vs 51.4% placebo · −1.2 ppInconclusive
n = 348 weighted
28 s
02 · Dustin Cathcart
352 actual respondents
Ballot support64.4%
vs 48.6% placebo · +15.7 ppDetectable
Comprehension42.3%
vs 51.4% placebo · −9.1 ppDetectable
n = 347 weighted
49 s
03 · Jesse Littlewood
352 actual respondents
Ballot support59.5%
vs 48.6% placebo · +10.9 ppDetectable
Comprehension55.4%
vs 51.4% placebo · +4 ppInconclusive
n = 347 weighted
43 s
04 · Paul Schilchtman
352 actual respondents
Ballot support61.4%
vs 48.6% placebo · +12.8 ppDetectable
Comprehension50.9%
vs 51.4% placebo · −0.5 ppInconclusive
n = 365 weighted
37 s
05 · Alex Rodriguez
352 actual respondents
Ballot support53.7%
vs 48.6% placebo · +5.1 ppInconclusive
Comprehension44.8%
vs 51.4% placebo · −6.6 ppInconclusive
n = 325 weighted
27 s
06 · Ajayi Harris
352 actual respondents
Ballot support52.5%
vs 48.6% placebo · +3.8 ppInconclusive
Comprehension54.7%
vs 51.4% placebo · +3.3 ppInconclusive
n = 375 weighted
Results are versus placebo. A difference between two point estimates does not establish a statistically significant head-to-head difference.
Two outcomes in detail
Which of the following best describes the main change the measure would make?
- Each party would keep its own primary, but independent voters could choose one to vote in
- CorrectAll candidates would run in one primary, and the two with the most votes overall would advance
- Primaries would be eliminated, and every candidate would go straight to the general election
- General elections would be decided by ranked choice voting
Subgroup results
Bars are levels — the % selecting Yes in one subgroup and arm — not lifts. Significant vs placebo inside that subgroup (90% criterion (two-sided two-proportion test)) Not marked Hatched, italic = source small-sample flag Placebo level in that subgroup
Political ideology and party ID
Education and 2024 presidential vote choice
What the study can and cannot establish
Supported by this study
- Randomized comparison of each video arm with the placebo arm on stated ballot choice and comprehension.
What it cannot establish
- Comparisons between video arms: each arm is tested only against the placebo.
- A verdict from any arm or subgroup with fewer than 50 actual respondents: those comparisons are inconclusive by rule.
What “no measurable effect” means
Inconclusive means the comparison did not meet the criterion; it is not evidence of zero effect.
Weighted counts by arm
| Placebo | 356 |
| Danielle Allen | 348 |
| Dustin Cathcart | 347 |
| Jesse Littlewood | 347 |
| Paul Schilchtman | 365 |
| Alex Rodriguez | 325 |
| Ajayi Harris | 375 |
Arm counts sum to the reported weighted total, 2,464.
Methodology notes
- Counted: 2464 complete records, the latest per panel id, env development. Not counted: 0 complete(s) from another env, 0 earlier duplicate(s), 0 with an arm outside the test, 0 screenout(s); 14 start record(s) seen.
- Raked to education, party, vote2024 in 4 iterations (largest remaining gap 9.7e-9); weights normalized to mean 1, range 0.27899999999999997 to 5.004, capped at 5 (23 capped). Targets: Massachusetts general adults (18+), derived from the audience store by scripts/results/targets.mjs.
- Weighting target, education (as raked): hs-or-less 31.2%, some-college 21.5%, bachelors-plus 47.3%. American Community Survey 5-year estimates (U.S. Census Bureau); education_4 @25, count; period 2020-01-01 to 2024-12-31; universe population 25+; proxy: population 25+; derivation audience-store. B15003 covers ages 25 and over; the store has no 18–24 education for the state, so the 25+ distribution stands in for adults 18+ and likely overstates bachelor's-or-higher for them.
- Weighting target, party (as raked): democrat 37.3%, republican 14.6%, independent-other 48.2%. Cooperative Election Study common content 2020, 2022, 2024 (Harvard University); pid3 @25, 2024, pct (commonweight); period 2024-01-01 to 2024-12-31; universe adults 18+ (CES state respondents, commonweight); derivation audience-store. Party identification, a survey estimate: n 1119, margin of error up to ±4 points. The printed shares sum to 100.1 and are renormalized to 100. Identification, not enrollment: the state's party enrollment measures registration, a different construct on a different universe. 2024 is used to match the 2024 vote target (--party-year picks another); the store also holds 2023 (n 383, ±5) and 2025 (n 272, ±6.3).
- Weighting target, vote2024 (as raked): democrat 37.4%, republican 22.0%, did-not-vote 38.2%, other 2.4%. Official ballot-measure results (state election offices); American Community Survey 5-year estimates (U.S. Census Bureau); presidential_vote @25, 2024, count; ACS age_bucket @25 (adults 18+); period 2024-11-05 to 2024-11-05; universe adults 18+; derivation audience-store. 3,512,930 ballots cast ÷ 5,679,875 adults 18+; "Did not vote" is the adults who cast no ballot (38.15%), non-citizen adults included. Blank presidential lines (a ballot cast, no presidential choice) count with "other" under the pax-entry map. Respondents over-report voting, so raking to the true non-vote share up-weights the non-voters a panel yields.
- Declared but not asked by this test, so not raked: age, gender, race.
- Levels are weighted percentages. Lifts are arm level minus placebo level, in points, with a margin of 1.645 standard errors. Each comparison is a two-sided two-proportion z-test with Kish effective sample sizes; detectable means p < 0.1 and at least 50 actual respondents in both groups.
Limitations
- Simulated respondents: the seeded demo run (scripts/results/seed-demo.mjs) walked the respondent flow, not a fielded panel. Every figure on this wave exercises the pipeline; none is a measurement.
- No multiplicity adjustment is applied across subgroups.
Source: pax-trinity-test-responses:ma-wave-3, computed 2026-10-29 (env development).