Synthetic Users: Merging Qualitative and Quantitative Research, in seconds.
At Synthetic users we are blurring the lines between qualitative and quantitative research. Here's how we are going about this transformative approach.
At Synthetic users we are blurring the lines between qualitative and quantitative research, presenting an exciting new avenue for data gathering and interpretation. Here's how we are going about this transformative approach, and what it means for the future of user experience and market research but more importantly to the future of business.
The equivalence of effort: Qualitative vs. Quantitative
Traditionally, qualitative research, characterized by interviews, focus groups, and case studies, has been viewed as labor-intensive and time-consuming. On the other hand, quantitative research, known for surveys, questionnaires, and numerical data, was seen as more scalable but sometimes lacking in depth.
The distinctions between qualitative and quantitative research were well defined by Creswell, in "Research Design: Qualitative, Quantitative, and Mixed Methods Approaches." He outlined how qualitative methods probe deep into human behavior, emotions, and motivations, while quantitative methodologies derive generalizable insights via statistical measures.
With Synthetic Users, this differentiation starts to blur. Running 10 interviews to gauge desirability or deploying 5,000 surveys can now demand roughly the same amount of effort.
Finding the sweet spot: the intersection of qual and quant
As we transition from strictly qualitative to more quantitative data sets, there lies an ideal intersection — a sweet spot where the depth of qualitative insights meets the breadth of quantitative data.
This point provides a balance, allowing researchers to obtain rich, in-depth insights while also leveraging the robustness of large-scale data. This equilibrium means that clients and stakeholders can then hone in on synthesized reports, gaining the benefits of both research types.
Synthetic Users allow for Nielsen's qualitative insights (even though Nielsen is not a fan of Synthetic Users) to coalesce with quantitative scalability. This merger leads to what Nielsen and Norman termed "triangulation", a comprehensive understanding of user needs and behaviors by employing multiple methods.
Nassim Taleb's"The Black Swan", makes sense here: "Having more data does not necessarily make you more knowledgeable, but it does make you more confident." The blend of qual and quant methodologies powered by Synthetic Users should, ideally, offer both — insight and confidence.
Data pioneer John W. Tukey, once remarked, "The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data." This emphasizes the importance of integrating both qualitative and quantitative methodologies. By doing so, powered by Synthetic Users, researchers achieve a more holistic and reliable understanding, ensuring both insight and confidence.
Understanding the mix: getting the best of both worlds
Think of Synthetic Users as tools that help us see things more clearly in a world where we humans give too much importance to a single story or experience, even if it's not common. Daniel Kahneman talked about this in his book "Thinking, Fast and Slow". Robert Cialdini did the same in "Influence: The Psychology of Persuasion," when he introduced the concept of scarcity where people attribute more value to opportunities when they are perceived as scarce.
By combining detailed stories (qualitative data) with numbers (quantitative data), researchers and anyone within the business can get a clearer, less biased picture. That is part of our mission, to clarify in order to potentiate better decision making.
When research involves humans, whether as subjects or as researchers, we are aware biases do emerge. Synthetic Users is able to mitigate the following:
Interaction biases are removed: In qualitative research, the very interaction between the researcher and the participant can introduce biases. For instance, a participant might provide answers they believe the researcher wants to hear (social desirability bias).
Observer-expectancy effect is neutralized: Sometimes, a researcher's expectations can subtly influence participants' behavior, leading to outcomes that align with the researcher's beliefs.
Sampling bias is adjusted by Synthetic Users to offer more diversity by default: Even the process of selecting participants can introduce bias. If a sample isn't representative of the broader population, the conclusions might be skewed.
Synthetic Users can sometimes overemphasize predefined user pains and goals. We're working to counteract this through dynamic interviews, where the emphasis is on study objectives over observed user pain points.
Synthetic Users are trained on vast amounts of data, and they generate outputs based on patterns in that data. If the training data has certain prevalent biases or if the questions posed to the model lead it in a particular direction, the model might produce outputs that appear biased or that emphasize certain patterns over others. We see this as a feature, not a bug — when we are delivering organic / synthetic parity.
Still, in order to mitigate this last bias a lot is being done.
Diverse Data: We ensure the training data of the models we use is diverse and well-representative in order to produce more balanced outputs.
Objective Questioning: we design prompts that offer up neutrality and don't necessarily lean towards any particular outcome.
Regular Evaluation: we periodically evaluate the performance and outputs of the model to check for any unintended patterns or biases.
So no, Synthetic Users don't have human-like biases, their outputs can still reflect biases present in their training data or in the way they're used. Any tool, regardless of how advanced, needs regular assessment and refinement to ensure it meets its intended purpose without introducing unintended biases.
A new more efficient world with better products and businesses
When deploying synthetic users in user experience and market research, combining the depth of qualitative insights with the breadth of quantitative data, we can anticipate a myriad of potential discoveries and advancements:
Richer User Profiles: We already notice these. By amalgamating vast datasets with detailed user experiences, Synthetic Users generate comprehensive user profiles that capture both broad behavioral trends and individual nuances. You will notice they are far richer than most outputs from organic research.
Predictive Insights: The union of qualitative and quantitative methods helps predict future user behaviors, preferences, and needs more accurately, facilitating proactive adjustments in product design or market strategy. This is what we are most excited about at Synthetic Users.
Refined User Journeys: With Synthetic Users mimicking potential real-world users, businesses can chart out intricate user journeys, identifying potential touchpoints, challenges, and opportunities for enhanced engagement. This is the area that needs more work on our part.
Hybrid Research Methods: We believe this blend might lead to the evolution of new research methodologies that harness the strengths of both qualitative and quantitative approaches, optimizing the research process.
Diverse Perspectives: Synthetic users, trained on a vast array of data, can simulate a broader range of user perspectives than might be feasible with traditional research. This will inevitably lead to products and services that are more inclusive and accessible.
Bridging Cultural Gaps: With access to global data, Synthetic Users can offer insights into cultural nuances, preferences, and behaviors, aiding businesses in tailoring their offerings for different geographies or demographics.
Speedier Iteration: The capability of synthetic users to rapidly process vast amounts of data and provide feedback means businesses can iterate designs, campaigns, or strategies faster, responding more agilely to market demands.
Deeper Emotional Understanding: While synthetic users don't possess emotions, their ability to analyze vast amounts of qualitative data can help businesses gain deeper insights into the emotional drivers behind user behaviors.
More than just skincare:
Scenario: A company selling skincare products receives thousands of online reviews monthly.
Action: Synthetic Users are trained with thousands of organic online reviews.
Result: The company finds that words like "confidence," "disappointment," or "relief" are frequently used. Synthetic Users indicate that they aren't just looking for skincare but for a boost in self-esteem. Products can then be improved with this emotional need in mind.
Airlines dealing with anxiety:
Scenario: An airline wants to improve its in-flight experience.
Action: Synthetic Users are created with real passenger survey data.
Result: Synthetic Users quickly reveal frequently mentioning feeling "anxious" about tight connections. As a response, they suggest that the airline introduces more reassuring notifications about connecting flights.
Evolving Ethical Considerations: As we deepen our use of Synthetic Users, there will be new discoveries in the realm of ethics, privacy, and data usage. How Synthetic Users are deployed, the data they access, and the insights they generate will necessitate ongoing ethical discussions and guidelines.
Using Synthetic Users in user experience and market research means we can combine deep, personal insights with big data. Join us as we explore this new approach. The sweet spot between qualitative and quantitative may yield new concepts that we are yet to discover.
We find it essential to balance both aspects and watch out for any issues. Doing this right will unlock the real benefits.
Releated Articles
More articles for you
The Lie We Tell Ourselves About Customer Research
Most research asks what people say. The problem is people don't do what they say. This piece breaks down the gap between stated and revealed preference — and why behavioral modeling, not better interviews, is how you close it.
Two ways to run research with Synthetic Users and why the difference matters
Iris, what is the difference of using agents to accelerate research.
Synthetic Users vs digital twins
You don’t need a twin for “a parent in rural Ohio who shops weekly at Walmart, prefers fragrance-free, and has a toddler with eczema.” You sample a parent profile with relevant traits and constraints, add retail and dermatology context, and generate behaviors consistent with both.
Two major papers. One shared direction.
LLM-powered Synthetic Users have crossed from concept to validated method. This proves they can predict human behavior accurately, letting teams run fast, low-cost behavioral experiments without replacing real participants.
Gartner says we lead. That's kind of them.
Gartner’s latest report on AI-powered synthetic user research cites Synthetic Users as a leader.
Introducing Shuffle v2
Shuffle v2 is a feature that intelligently shuffles between multiple large language models via a routing agent to produce more realistic, diverse Synthetic Users with better organic parity.
Chain-of-feeling
Synthetic Users use a “chain-of-feeling” approach—combining emotional states with OCEAN personality traits—to produce more human-like, realistic user responses and yield richer UX insights.
Generative Agent Simulations of 1,000 People
A paper that thoroughly executes a parity study between Synthetic and Organic users.
21 Peer reviewed papers that support the Synthetic Users thesis
Here is a compilation of all the papers that help make a case for Synthetic Users.
Why we shuffle between models — to ensure both parity and diversity!
Synthetic Users balances aligned and unaligned models to maintain diversity and authenticity in simulated users while ensuring ethical standards and user expectations are met.
Latest press articles for Synthetic Users
Synthetic Users and AI are transforming research methodologies, offering innovative, cost-effective alternatives to traditional human subject studies.
Comparison studies. The opportunity lies in the deviation.
When we compare different studies, especially looking at what synthetic (artificial interviews) and organic (real-world interviews) data tell us, we often find they mostly talk about the same things but there's also a bit where they don't match up. This gap is super interesting because it's like finding hidden treasure in what we thought we knew versus what we might have missed.
How we deal with bias
Harnessing the power of AI in our Synthetic Users, we strive for a balance between reflecting reality and ethical responsibility, ensuring diversity and fairness while maintaining realism.
The transition to Continuous Insight
The transition towards Continuous Insight™ aligns research activities more closely with the dynamic needs of the business and ensures that product development is continuously informed by up-to-date user insights.
The Art of the Vibes Engine
Large language models (LLMs) like GPT-4 serve as powerful "vibes engines," empathizing with diverse groups and generating contextually relevant content. Their applications span market research, customer support, user experience design, and mental health support, offering invaluable insights and personalized experiences. While not infallible sources of truth, LLMs enable creativity, personalization, and connection within the realm of human language.
There is a faster and more accurate way to do research. Use Synthetic Users.
How Synthetic Users is changing the research process.
The wisdom of the silicon crowd
In the light of an ancient parable, we explore a new paper that dives into how ensembles of large language models match the prediction accuracy of human crowds. It reveals that combining machine predictions with human insights leads to the most robust forecasting results.
Three research papers that helped us build ❤️ Synthetic Users
For the sceptics amongst us who need more tangible research in order to engage with this brave new world. Full disclosure: we are part of the sceptics.
What is RAG and why it’s important for Synthetic Research
Ahead of our RAG launch we explain Retrieval-Augmented Generation (RAG) and how it enhances Synthetic Users by providing increased realism, contextual depth, and adaptive learning, with profound implications for market research, user experience testing, training, education, and innovative product development.
Synthetic Users system architecture (the simplified version).
Foundation models underpin Synthetic Users with advanced capabilities, enhanced by synthetic data and RAG layers for realism and business alignment, all within a collaborative multi-agent framework for richer interactions.
Saturation score. How do we know how many interviews to run?
Determine your interview target for achieving topic saturation using our efficient approach, leveraging the historical wisdom of research pioneers. This method ensures deep insights with theoretical sampling at its core.
How Synthetic Users are gaining depth
Synthetic Users are evolving to address criticism about their generalist nature by incorporating representative data sets and personal narratives.
How we compare interviews to ensure we improve our Synthetic Organic Parity — 85 to 92%
How do we know we are right? How do we know our Synthetic Users are as real as organic users? We compare.
Synthetic Users: Merging Qualitative and Quantitative Research, in seconds.
At Synthetic users we are blurring the lines between qualitative and quantitative research. Here's how we are going about this transformative approach.