AI Panels: The Illusion of Sample Accuracy and Precision

One of the core principles of market research is that your sample needs to do two things well: be precise (tight enough that you can trust the estimate isn’t noise) and accurate (centered on the truth about your target population). Synthetic respondent panels compromise both, and we wish vendors were more transparent about this.

The Precision Problem

In traditional research, increasing your sample size makes your estimates more precise. The confidence intervals get tighter, and the margin of error shrinks. This works because each additional respondent is an independent observation drawn from a real population, and each new person adds new information.

Synthetic panels don’t work this way. When a vendor scales a panel from 500 to 5,000 respondents, they are running the same model on the same prompt with slight variations in internal randomness. Each output is generated by the same underlying system, drawing on the same learned patterns. The responses are not independent, and never will. That’s because they all originate from a single source – the underlying neural network model.

This matters because every inferential test researchers use  (e.g. chi-square, t-tests, ANOVA, regression) assumes that observations are independent of one another. When that assumption holds, adding more observations genuinely increases statistical power. When the assumption doesn’t hold, adding more observations just produces tighter numbers around an estimate that was never grounded in real variation.

Here is a practical way to see it. In traditional research, if you draw two independent samples of 500 from the same population, you will see natural variation between them. The distributions won’t be identical, because the people are different. Run the same prompt twice on a synthetic panel and the distributions will converge, because they are generated by the same model. That convergence is the statistical signature of a system repeating itself, not sampling.

The confidence intervals look tight. The margin of error looks small. But the precision is artificial. It reflects the consistency of a model, not the convergence of independent observations toward a real population value.

The Accuracy Problem

Precision tells you how tightly your estimates cluster. Accuracy tells you whether they cluster around the right answer. Even if synthetic panels could deliver real precision (which they don’t), the accuracy problem would remain.

The problem with accuracy in synthetic responses is that they aren’t accurate relative to a population or anything remotely real. In fact, accuracy is nothing more than the exact wording used in the AI prompt – and variations depend on the skill (and trust) of the engineer who designed your synthetic panel software.

Synthetic respondents are not drawn from a population. They are generated by a model trained on existing data and social media observations, and then prompted to simulate people with certain characteristics. What determines the output is not the characteristics of real people. It is the patterns in the model’s training data and the wording of the prompt.

Change the wording of a demographic description in the prompt and the response distributions shift. Not because the population changed, but because the instruction changed. In real research, rephrasing a screener doesn’t change who qualifies. In a synthetic panel, it changes everything, because there is no population. There is only the prompt.

Vendors will point to validation studies showing that synthetic panels match real panels on top-line distributions. This sounds reassuring. It isn’t. A model trained on real survey data will naturally reproduce the central tendencies of that data. Matching the average is memorization, not validation. The question is whether the relationships between variables, the tails of the distributions, and the unexpected patterns that drive real insight are preserved. They aren’t, because the model learned aggregate tendencies, not the individual-level variance that creates those patterns.

In traditional research, accuracy comes from sampling real people from a defined population. The responses reflect actual opinions and actual behaviors from humans with real context. When you calculate a mean or a proportion, it is an estimate of something that exists in the real world.

What This Means

A synthetic panel gives you tight confidence intervals around an answer that have more to do with the neural network parameters than your demographic. The precision is artificial and the accuracy unverifiable. And the combination is worse than either problem alone, because the output looks authoritative… which leads executives to make bad decisions.  

As a trained researcher, it’s vital that you hold the line between polling a sample of human beings, and asking an unknown AI model what it thinks your respondents should say.