The Sample Size Illusion: Why More Synthetic Data Isn’t More Information

When you are vetting research vendors, you will often see them lead with sample size. Thousands of synthetic respondents, delivered in minutes, at a fraction of the cost of a real panel. The pitch is built around a number that every researcher has been trained to respect: a larger N. A larger N means more statistical power, tighter confidence intervals, and more reliable estimates… but only when you’re sampling real people. It is not true when you’re generating synthetic responses.

More rows of synthetic data do not equal more information. They equal more repetition.

One Model, One Logic

In traditional research, every human respondent is an independent data source. They bring different life experiences, different decision-making contexts, different biases, and different moods on the day they took the survey. This variation is the foundation of inferential statistics, and is why a larger sample gives you more statistical power. 

Synthetic data works differently. Every single “respondent” in an AI panel is generated by the same underlying model, trained on the same data, and operating under the same learned patterns. When a vendor runs that model 5,000 times with slight variations in internal randomness, they are not interviewing 5,000 individuals. They are asking a single algorithm the same question 5,000 times and collecting the variations in its output.

The outputs will indeed differ from one another. The model introduces small amounts of randomness in how it selects words and constructs responses. Vendors point to this variation as evidence that each “respondent” is distinct. But this is cosmetic. It is surface-level noise generated by the same system, with an model parameter called “temperature” adding variation. The underlying logic, the learned patterns, and the statistical tendencies that shape every response are identical across all 5,000 outputs.

This is why your confidence intervals shrink when you scale a synthetic panel. Not because you are converging on a true population parameter, but because a single model is producing increasingly consistent output. The estimate looks precise. But precision driven by a model repeating itself is not the same as precision driven by independent observations converging on reality.

In statistical terms, the effective sample size of a synthetic panel is far smaller than the row count suggests. Possibly as small as one, because every observation traces back to the same source. The N on the output is a count of model runs. It is not a count of independent data points. Every statistical test you run on that data inherits this problem, and none of them will flag it for you.

A Veneer of Insight

Synthetic panels are appealing because they deliver on the three things every project manager cares about: speed, scale, and cost. A vendor can generate thousands of responses overnight for a fraction of what a real panel costs. For organizations under pressure to produce insights faster and cheaper, that pitch is hard to ignore.

But those promises come with a tradeoff that isn’t disclosed in the vendor demo. The data looks complete. The cross-tabs populate. The charts render. Everything has the appearance of a finished analysis. What’s missing is the substance underneath it.

For example, think of the example of running a segmentation on synthetic data. The clusters will form, because the algorithm will always find clusters. But those segments do not contain the kind of variation that makes segmentation useful in the first place. This is an exercise in futility because you are clustering output from a single entity – the AI. Any clusters that are found are simply the randomness of temperature inserted. You’re measuring the differences in noise, not unexpected combinations of attitudes and behaviors.

The same problem shows up in forecasting, driver analysis, or any technique that depends on the relationships between variables being authentic. You are analyzing a reflection of what the model learned during training, not a measurement of what is actually happening in the market. The output will always look reasonable. It will rarely be surprising. And in research, when the data never surprises you, that is a sign that something is wrong with the data.
Real respondents can change their minds, contradict themselves, behave irrationally, and reveal patterns that no model would predict. That messiness is not a flaw in human data. It is the signal. Synthetic panels eliminate it by design.

Use it, Don’t Rely on it

None of this means synthetic tools are entirely without value. In narrow, exploratory contexts they can serve a purpose. Testing survey language before fielding. Brainstorming potential category structures. Generating placeholder data to prototype a dashboard layout. These are tasks where the output doesn’t need to be statistically valid, because no decision is being made on the basis of the results.

The danger is when synthetic data moves from exploration to evidence. When it becomes the basis for a segmentation strategy, a pricing decision, a campaign brief, or a board presentation. At that point, the illusion of sample size becomes a liability, because every conclusion carries a confidence it hasn’t earned.

As a researcher, you know the difference between data that was measured and data that was generated. Your stakeholders may not. Part of the job now is making sure they understand that a large N from a synthetic panel is not the same as a large N from a real one. The number may be the same. What it represents is fundamentally different.

Insights that move the needle for your organization need to be grounded in evidence that was observed, not fabricated. A smaller sample of real people telling you something unexpected will always be worth more than a massive synthetic panel confirming what the model already believed.