Science
The parity stack, explained with a flu shot
A parity score is only as good as the test behind it. Five layers make a Synthetic Organic Parity test hard to argue with. Here they are, in one simple healthcare example.
Every number in this example is illustrative. We made them up to show the method. They are not results from a client study.
§ 1The example: a flu shot reminder
A regional health system wants more adults over 60 to book a flu shot. It has three text messages to choose from.
- Message A. "Flu can put you in hospital. Protect yourself."
- Message B. "Keep your grandchildren safe this winter."
- Message C. "Free. Ten minutes. No appointment needed."
The question is simple. Which message gets the most bookings?
The organic way: survey 400 patients. It takes weeks.
The synthetic way: ask 400 synthetic users built to match the same audience. It takes an afternoon.
Before the health system trusts the afternoon version, it wants proof. Here is that proof, one layer at a time.
§ 2Layer 1: Holdout
The rule: synthetic users never see the organic answers.
The health system surveys 400 organic patients. Then it seals the results.
We receive two things only: the audience definition and the questionnaire. The synthetic users answer. Only then is the seal broken.
Here is what the health system sees when it opens the seal.
| Preferred message | Organic | Synthetic |
|---|---|---|
| A. Protect yourself | 31% | 36% |
| B. Grandchildren | 45% | 40% |
| C. Free and fast | 24% | 24% |
Both groups pick Message B. The agreement score is 95%.
If the synthetic users had seen the organic answers, a match would prove nothing. It would be a student with the answer key. If a client wants to share past data for calibration, split it. Share half. Lock half. Test only against the locked half.
§ 3Layer 2: Blind scoring
The rule: the judge does not know which side is which.
The survey also asks why. Those open answers become themes. A researcher at the health system receives two theme lists, labelled Set 1 and Set 2. Nothing says which one is synthetic.
Five of six themes match.
Only organic patients raised getting to the pharmacy.
Only synthetic users raised cost.
We report the misses, not just the matches.
People see what they hope to see. A scorer who knows which list is synthetic will grade it kindly or harshly. Removing the labels removes the bias.
§ 4Layer 3: The organic ceiling
The rule: measure how much organic people agree with each other first.
Two groups of organic people never agree 100%. So 100% is the wrong target.
The health system surveys a second group of 400 organic patients. Same questions. Same audience.
| Preferred message | Organic, group 1 | Organic, group 2 |
|---|---|---|
| A. Protect yourself | 31% | 33% |
| B. Grandchildren | 45% | 43% |
| C. Free and fast | 24% | 24% |
The two organic groups agree at 98%. That is the ceiling.
Without a ceiling, nobody can tell if 95% is excellent or mediocre. With it, the claim becomes precise. Synthetic users landed three points below what a second organic study would have delivered.
§ 5Layer 4: Outcome prediction
The rule: predict what people do, before they do it.
Survey answers are what people say. Bookings are what people do.
Before the campaign starts, the synthetic users predict the booking rate for each message. The prediction is written down and sealed.
Then 30,000 patients receive a text. 10,000 get each message. The health system counts the bookings.
This is the strongest layer. Nobody could have seen the answer in advance. It did not exist yet.
§ 6Layer 5: The client runs it
The rule: the buyer holds the keys.
Look back at who did what.
- The health system recruited the organic patients.
- The health system held the seal.
- The health system chose the scorer.
- The health system counted the bookings.
We supplied the synthetic side. Nothing else.
A parity score we grade ourselves asks for trust. A parity score you grade yourself needs none.
§ 7Keep it fresh
Once a holdout is opened, it is seen data. It cannot be used as a holdout again.
Next flu season, the health system repeats the test with new patients and a new seal.
Parity is a habit, not a certificate.
§ 8The stack at a glance
| Layer | Question it answers | In the flu shot example |
|---|---|---|
| 1. Holdout | Were the answers hidden? | Organic survey sealed until synthetic users finished |
| 2. Blind scoring | Was the judge fair? | Theme lists labelled Set 1 and Set 2 |
| 3. Organic ceiling | How high can the score go? | Two organic groups agreed at 98% |
| 4. Outcome prediction | Did it predict what people did? | Sealed forecast, then 30,000 texts |
| 5. Client runs it | Who graded the test? | The health system, on its own data |
Hide the answers. Blind the judge. Measure the ceiling. Predict the outcome. Hand over the keys.
Notes
- The agreement score used here is 100 minus half the sum of the absolute gaps between two sets of shares. For organic versus synthetic: gaps of 5, 5 and 0 points give 100 − 5 = 95. For the two organic groups: gaps of 2, 2 and 0 points give 100 − 2 = 98.
- The average miss in Layer 4 is the mean absolute error across the three messages: 1, 2 and 1 points, or 1.3 points.
- A cheaper way to measure the ceiling is to split one organic sample into two random halves. If you do, size the synthetic panel to match one half, so every comparison uses the same sample size.