Last updated: 1 September 2026
A general-purpose assistant is a good desk researcher and a poor respondent. It will summarize a category, draft a screener, restructure a discussion guide and pull themes out of transcripts you already own. All of that in minutes. What it cannot do is tell you what people who are not in the room think, because nothing in the system has asked them.
That boundary sounds obvious stated plainly, and it is crossed constantly. The crossing usually looks like a reasonable shortcut: rather than fielding a study, someone asks a model to describe how a 34-year-old parent in Columbus would react to a proposition, and the answer arrives fluent, specific and confident. Fluency is not evidence, and the research literature on this is now unusually direct about where the failure sits.
The useful framing is not whether assistants are good or bad at research. It is which parts of a research project are made of existing text, and which parts are made of people. Assistants are transformative on the first and inert on the second.
Where Does a General Assistant Genuinely Help?
The honest list is longer than most research vendors admit.
- Desk research and category orientation. Summarizing published material, building a competitive landscape, finding the terminology a category actually uses.
- Instrument drafting. A first-pass screener, discussion guide or survey, which a researcher then rewrites. The draft is rarely good and it is much faster than a blank page.
- Synthesis of material you already own. Coding themes across transcripts, past decks and support tickets is a text problem, and text problems are where these systems are strongest.
- Translation and localization drafting, with native review before anything reaches a respondent.
- Analysis assistance. Restating a finding for a different audience, drafting the executive summary, checking whether a chart supports the sentence above it.
- Stress-testing your own reasoning. Asking what would have to be true for a conclusion to be wrong is a genuinely good use of the tool.
Every item on that list operates on material that already exists. That is the pattern, and it is the whole boundary.
Why Desk Research Is Not Primary Research
Desk research tells you what has already been published about a market. Primary research tells you what your specific customers think about your specific proposition, which by definition has never been written down.
The distinction gets blurred because assistant output reads like research. It has structure, hedged claims, apparent sourcing. But a synthesis of public material inherits every gap in that material, and the gaps are not random: they concentrate in categories that are under-researched, in markets that publish less, and in populations that surveys already miss.
Where the published record is thin, the output is not thin. It is confident, which is worse.
The operational test is simple. If a stakeholder asked "how do you know?", could you name a person who was asked? If the answer traces back only to published text, you have done desk research, and it should be labeled that way in the deck.
What Happens When You Ask a Model to Be Your Respondent?
This practice has a name in the literature, silicon sampling, and it has been evaluated enough to say something concrete about where it holds and where it fails.
The consistent finding is that simulated responses track population-level averages considerably better than they track subgroups. Work published in Political Analysis on synthetic replacements for human survey data sets out the perils directly. An arXiv evaluation of survey responses generated by large language models examines how far those responses can be treated as survey data at all.
Two findings matter for anyone considering this as a cost-saving measure:
- Error is unevenly distributed across groups. Evaluations report that simulation errors concentrate in particular demographic and political segments rather than spreading evenly. The technique is least reliable precisely where segment differences are the thing you are trying to measure.
- Population-level estimation and individual-level simulation are different problems. Statistical correction can improve aggregate estimates without solving individual simulation, because the latter requires capturing idiosyncratic traits the model has no access to. A broader review of large language models as synthetic social agents makes the same separation.
The commercial reading: if you already know the population average, a simulation can approximate it. If you need to know what a specific segment thinks and why, which is the reason most consumer research is commissioned, the technique fails at exactly the point of interest.
Which Research Tasks Are Safe to Automate?
| Task | Automate with an assistant? | Why |
|---|---|---|
| Category desk research | Yes, with source checks | Operates on published text; verify anything load-bearing |
| Drafting a screener or guide | Yes, then rewrite | Speeds the first draft; a researcher still owns the logic |
| Coding themes in your own transcripts | Yes, with human audit | Text you collected; audit a sample against the source |
| Translating an instrument | Draft only | Needs native review before fielding |
| Estimating market size | No | Compounds published estimates without exposing their assumptions |
| Generating respondent answers | No | See the section above; fails hardest at subgroup level |
| Deciding what to launch | No | Requires evidence from people who might buy it |
Best case for full automation: coding themes across transcripts you already collected, where the source material is yours, the output is auditable against it, and the alternative is a researcher doing the same job more slowly. That is a real productivity gain and it is worth taking.
The row worth arguing about is market sizing, because it is the one buyers most often assume is safe. An assistant asked to size a category will produce a number by combining published estimates, and it will not surface that those estimates were themselves derived from each other, or that the definitional boundaries differ between them. The output is a figure with no error bars and no visible lineage, which is considerably more dangerous than an honest refusal.
If a sizing number is going into a business case, the assumptions behind it have to be legible. That means building it yourself from sources you can name. The same caution applies before concept testing, where a sizing error upstream quietly sets the wrong bar for every concept that follows. What usability work costs and includes is covered in usability testing services.
Who Is Missing When the Evidence Is Text?
A model trained on published text inherits the composition of what gets published, and that composition is not the composition of a market.
The gap is measurable at the connectivity layer before it reaches the language layer. Pew Research Center's mobile technology fact sheet reports 16 percent of US adults as smartphone-only internet users. That rises to 34 percent in households under $30,000 a year against 4 percent above $100,000.
The ITU's connectivity statistics show how much wider that gradient runs globally. Pew puts overall smartphone ownership at 91 percent of US adults, so the divide is broadband and privacy rather than handsets.
People who are online through a phone, in a language with less published text, generate less of the record a model learns from, and are correspondingly less well represented in anything derived from it.
That is the structural reason the substitution is tempting and wrong. The populations hardest to reach through conventional fieldwork are also the populations least represented in training data, so the shortcut degrades most exactly where the fieldwork was hardest and the answer mattered most.
Reaching them is a channel problem rather than a modeling problem. Alchemic runs interviews natively inside WhatsApp with no link and no app to install, and by outbound phone call for respondents who are reachable by voice and not by browser. Alchemic publishes 57+ languages including Spanish, Hindi, Tamil, Bangla, Arabic and Indonesian. The same selection effect is examined in sample validity and who you miss.
Recruitment runs as managed fieldwork or bring your own. The point is not that automation is avoided; it is that the respondent is a real person who was actually asked. When that data holds up is examined in when AI-moderated interviews produce reliable data.
Where AI-Moderated Interviews Sit Between the Two
There is a middle category that gets confused with both ends, and the distinction is worth holding.
An AI-moderated interview is not a model generating answers. It is a model conducting a conversation with a human respondent, asking follow-up questions based on what that person said, and recording their actual words. The evidence is human; the moderation is automated. That places it with primary research on the question that matters, which is whether anyone was asked, and with automation on cost and scale.
How much latitude such a system has to leave the discussion guide varies considerably between tools, and it is the right thing to probe in a pilot rather than assume. A system that is rigid about the guide will hold the script when the interesting answer is one question sideways; a more dynamic one treats the guide as a starting point. Where the study runs as a managed service, researchers design and tailor the guide to those risks from the brief before fielding rather than leaving that work to the buyer.
The overview of AI-moderated interviews covers where depth and scale trade against each other.
For governance vocabulary around any of this, the NIST AI Risk Management Framework is vendor-neutral and specific. The ESOMAR code and guidelines set the professional expectations for disclosure and respondent treatment that apply regardless of how the interview is moderated. Where a human moderator still wins is worked through in AI against human moderated interviews.
Where This Comparison Breaks Down
Four limits on the argument above.
The capability boundary moves. Evaluations describe systems at a point in time. Anything asserted here about what models cannot do should be re-tested rather than assumed permanent, and the Stanford HAI AI Index is a reasonable neutral place to track how fast the measured picture changes.
"Not primary research" is not "useless". Desk synthesis is genuinely valuable and is routinely undervalued by researchers defending their craft. The failure mode runs both ways.
Human research has its own validity problems. Survey satisficing, social desirability bias and panel professionalization are real and well documented. The comparison is not perfect evidence against synthetic evidence; it is two imperfect methods with different failure modes.
Small qualitative samples do not become quantitative because a machine processed them. Published work on saturation in qualitative interviewing, including a systematic assessment of thematic saturation, found saturation at six interviews in two of three datasets and at eight to nine in the third. Automating the moderation does not change what a sample of that size can support, and reporting it as a percentage of a market remains wrong however it was collected.