Last updated: 17 September 2026
Quick Answer: Probability and non-probability sampling differ on one rule: only probability sampling gives every unit in the frame a known, non-zero chance of selection. That difference licenses a margin of error, and AAPOR found the standard calculation does not transfer. Probability methods are simple random, systematic, stratified, cluster and multistage; non-probability methods are convenience, quota, purposive, snowball, river and opt-in panels.
In December 2023 an online opt-in poll put agreement that the Holocaust is a myth at 20% among US adults under 30. On a probability-based panel recruited by mail, Pew Research Center asked the same question in January 2024 and measured 3%. The gap is about who ended up in the sample.
What Changes When Every Unit Has a Known Chance of Selection?
Known, non-zero selection probabilities are what let you report a margin of error.
A September 2024 expert brief for the US Department of Health and Human Services, by Michael Bailey and J. Michael Brick, defines probability sampling as selecting units so each in a finite population has a known, positive probability of selection. Nonprobability sampling, the closed-up federal spelling, is any approach without them.
The American Association for Public Opinion Research task force reporting in June 2013 was blunt: the standard margin of error calculation is unsuitable for non-probability samples, and treating estimates as error-free is not reasonable either. It also advised replacing response rate with the ISO 20252 term participation rate, the denominator for a traditional response rate not always being known.
What Are the Probability and Non-Probability Sampling Methods?
Five probability methods are standard.
- Simple random. Every unit in the frame is equally likely, which needs an enumerable frame.
- Systematic. A random start, then every kth unit in an ordered frame.
- Stratified. A random sample drawn inside each group of the frame.
- Cluster. Whole groups drawn at random, units inside them surveyed.
- Multistage. Selection in stages, combining the schemes above.
Six non-probability methods cover almost everything commercial. In none can you state a person's selection probability.
- Convenience. Whoever is reachable and willing.
- Quota. Cells set on observable variables and filled until each closes. The easiest design to mistake for a probability one: the cells look like strata, but nothing inside one is random.
- Purposive or judgment. Chosen for what they should know or represent.
- Snowball and respondent-driven. Heckathorn's 2011 Sociological Methodology paper separates the two: plain snowball is a convenience method, while respondent-driven sampling adds seeds, recruitment matrices and network-size controls, and yields bounded intervals.
- River or intercept. Respondents caught mid-session on someone else's inventory.
- Opt-in online panels. People join, are invited and are screened in.
How Do Quota and River Samples Work?
A beverage concept test, 400 interviews. Quotas on age, gender and region fill from an online panel inside 36 hours and every cell closes. What you hold is a sample of panel members who opened the invitation early and passed a screener, weighted on three variables.
An ad test on river sample is the other shape: respondents intercepted on publisher inventory mid-session, no panel membership and no frame. Selection bias starts in sourcing rather than in the questionnaire, and how respondents are recruited and verified is its own subject.
The eleven methods group by family, probability first, alphabetically within each.
| Method | Family | What the result supports |
|---|---|---|
| Cluster | Probability | Design-based estimates, computed error |
| Multistage | Probability | Design-based estimates, weighted |
| Simple random | Probability | Design-based estimates, margin of error |
| Stratified | Probability | Design-based estimates, by group |
| Systematic | Probability | Design-based estimates unless the order is periodic |
| Convenience | Non-probability | The people reached, nothing wider |
| Opt-in online panel | Non-probability | Model-based, conditional on weights |
| Purposive (judgment) | Non-probability | Depth on a defined case |
| Quota | Non-probability | Model-based, margins matched |
| River (intercept) | Non-probability | Fast reads, no participation rate |
| Snowball or respondent-driven | Non-probability | Hidden populations, bounded under RDS |
Can You Use Both Probability and Non-Probability Sampling in One Study?
Yes. The usual shape is a small probability-based reference survey supplying the targets and a larger, cheaper non-probability sample supplying the volume, joined by a weighting or modeling step. It buys better calibration, not a probability sample.
What keeps a hybrid honest is the task force's own conclusion: inference from any survey, probability or not, rests on modeling assumptions, which belong in front of the reader.
Why Do Most Commercial Consumer Studies Run on Non-Probability Samples?
Because the frame does not exist, building one costs more than the decision, and the telephone that used to substitute for it stopped working. The collapse in telephone survey response rates removed the one general-population frame commercial buyers could field against cheaply.
How Do You Judge One You Cannot Randomize?
By what the provider discloses and by what the estimates do when tested.
- Frame and coverage. Which population, through which channel, and who does that channel exclude?
- Recruitment sources. Named sources, each one's share of the delivered sample, and a participation rate.
- Quota and weighting disclosure. Which variables, which targets, which benchmark.
- Replication across waves. Same instrument and source again; movement beyond the reported interval is a model artifact.
- Fraud controls, and their limits. In Pew's August 2026 methods study of 11,114 opt-in respondents, trap questions flagged 18% of cases and a prescreening service failed near half of all completes. Voter-file matching removed 64% and raised error by discarding valid respondents. No screen is a fix.
The question lists exist: ESOMAR's 37 Questions to Help Buyers of Online Samples and AAPOR's 2023 report on data quality metrics for online samples. Alchemic runs managed fieldwork or bring your own, so a study's source is its own panel network, the client's lists, or a hybrid top-up, with live screening and quotas, automated fraud flags and audit trails. A provider who cannot name the source, quota targets and fraud controls has failed checks two, three and five.
Who Gets Left Out of Both Designs?
Coverage error is the failure both families share. A probability design is only as good as the frame it enumerates, and a non-probability design inherits the coverage of whatever channel recruits for it.
ITU's Facts and Figures 2025 puts 2.2 billion people offline, most in low- and middle-income countries, with mobile broadband coverage nearly universal while quality and affordability gaps persist.
Changing the channel changes the frame. Alchemic runs interviews natively inside WhatsApp with no link and no app and as outbound AI phone calls. It publishes 57+ languages including Spanish, Arabic and Mandarin, and fields across 14 markets including the USA and the UK. Treating a reach gain as a validity gain is how a study ends up defending the wrong thing.
Where This Distinction Stops Being Useful
Purposive sampling is the correct method for small-n qualitative work, and a probability design would be the wrong tool. Campbell and colleagues, in the Journal of Research in Nursing in 2020, judge a purposive sample on credibility, transferability, dependability and confirmability instead of sampling error.
The probability label describes the selection step, not the realized sample. Bailey's half of the 2024 federal brief is direct: a random contact survey does not necessarily produce a random sample once response rates are very low. Probability-based sampling does not eliminate error there, but it attenuates error and stays the best option available, which is narrower than the textbook version and truer.
Probability sampling earns the right to publish an interval; everything else earns the right to be checked, replicated and disclosed, which for most commercial decisions was the working standard anyway.