November 22, 2024
Generative Agent Simulations of 1,000 People
A paper that thoroughly executes a parity study between Synthetic and Organic users.
Comparison studies. The opportunity lies in the deviation.
When we compare different studies, especially looking at what synthetic (artificial interviews) and organic (real-world interviews) data tell us, we often find they mostly talk about the same things but there's also a bit where they don't match up. This gap is super interesting because it's like finding hidden treasure in what we thought we knew versus what we might have missed.
We run comparison studies where there is a huge overlap in topics but then there is also a deviation. In early 2024 we're at an average of 15% deviation between Organic and Synthetic interviews. That’s where we are. You can approach this deviation in two ways.
- Work that needs to be done to get full parity. That is our job and we're on it.
- An opportunity to explore themes or topics that organic users failed to mention.
‍
See below how we map Synthetic Organic parity through manual theme extraction when we run comparison studies:
The overlap is considerable but then there is a deviation. See below: topics exclusively mentioned by Synthetic Users (Purple) as well as Organic users (Blue)
One way to leverage this deviation is by conducting further research to understand the reasons behind the disparity. By diving deeper into why organic users failed to mention certain topics, we  gain valuable insights and identify potential areas of improvement. Additionally, this deviation can serve as a creative launchpad for exploring new themes or topics that have not been mentioned by either group of users.
‍
‍