Science
How we deal with bias
An LLM on its own is a biased narrator. Our job is to know exactly how.
In September, Pew Research had frontier LLMs answer nearly 300 of its survey questions.
On average, the AI was off by 12 points.
For Republicans and Black respondents, it was off by more than 15.
Organic respondents were four times more likely to say "not sure."
We agree with Pew.
An off-the-shelf model is not a panel.
But asking whether LLMs are biased isn't useful. They are.
The useful question is which biases to remove and which to keep.
Organic people are biased too.
A synthetic user with no bias wouldn't be realistic.
So we sort bias into two kinds.
1. Model bias: we correct it
Model bias is when the LLM distorts the world.
It lowers Synthetic-Organic Parity, so we remove it.
These are the ones we see most.
The Paris problem
Ask an LLM for 10 synthetic users in France, and most of them live in Paris.
That region holds less than a fifth of the population.
Stanford researchers found subtler geographic bias too.
Models rated places in poorer regions lower on attractiveness, morality and intelligence.
Our default now follows the actual population spread.
The saint problem
LLMs want to be liked.
A Stanford study found that within a few questions, models recognize a personality test.
Then they drift toward the most socially desirable answers.
We saw this as eco bias: synthetic users ready to save the planet at any cost.
Organic people say that too, but their behavior tells another story.
We calibrate for what people do, not only what they say.
The fan problem
LLMs like everything.
A 2026 study of more than 550,000 silicon respondents found them "super-omnivorous", with a systematic bias toward liking.
In concept testing, that's dangerous: every idea looks like a winner.
We anchor preferences to organic benchmarks, not to the model's enthusiasm.
The certainty problem
Organic people hesitate, and models don't.
Indifference and confusion are findings, not noise.
We keep them.
The hype problem
For a while, every solution our synthetic users proposed involved VR or crypto.
It was funny at first. Then we compensated.
The stereotype problem
This one is the subtlest.
LLMs can flatten identity groups into caricatures, as Wang and colleagues showed in Nature Machine Intelligence.
A paper from October found that silicon samples match the answers but not the respondents.
The averages look right. The individuals don't.
Simple statistical models fitted to organic data beat every LLM it tested.
That's close to how we work.
We build populations with IPF and copula sampling.
Then the LLM gives each synthetic user a voice.
2. Societal bias: we keep it, and make it visible
Some bias isn't a model error. It comes from us.
In a University of Washington study, AI résumé screeners favored white-sounding names 85% of the time.
They favored Black-sounding names less than 10% of the time.
Black male names were never favored over white male names.
The models learned this from us.
In a hiring tool, that mirror does harm.
It also raises a harder question.
As more of the world's text is written by AI, will AI lock our biases in?
Our position:
Synthetic Users is a research tool, not a tool that makes decisions.
To read a market accurately, we reflect society as it is. No more, no less.
But reflecting a bias is not the same as hiding it.
In our ideal tool, every bias is visible.
You decide, with eyes open, whether to correct for it.
3. How we catch what we can't see
Bias doesn't always show up in the averages.
A 2026 University of Maryland study looked at nearly one million LLM-written résumé summaries.
It found bias in the tails: the summaries read the same across names, yet still made hiring decisions less stable.
So we test four ways:
Organic benchmarks. We check synthetic results against organic data and publish the scores on our live parity dashboard.
Model diversity. Interviews run across an ensemble of LLMs. When they disagree, one of them is usually biased.
Counterfactuals. We swap one detail, like a name, and see what moves. It's the same method the UW team used.
Trust, per study. As one recent paper puts it, the question is when you can trust your synthetic users. We answer it with parity scores, study by study.
Diversity is how you move past the mirror
You might see LLMs as mirrors, or as a chance to correct injustice.
Either way, you can't correct what you can't see.
So first, an accurate mirror.
Then diverse data, to capture the full range of human behavior and offset the skews models learned.
Not less realism.
Realism with your eyes open.