Sample Size vs. Representativeness in Polls

Ask how respondents entered

A million visitors to one community can still differ systematically from the population named in a headline. Recruitment defines whose preferences are visible.

Map the path into the sample: who was eligible, who could be reached, who noticed the invitation, and who chose to respond. At each step, ask which people may be missing or unusually motivated. This path defines the group the data can describe. A large final count does not restore people excluded earlier in recruitment.

Distinguish precision from coverage

More responses can stabilize the percentage among participating users. They do not add people who never saw the poll or correct a reason that one group responds more often.

More responses can reduce random fluctuation inside the observed participant process, but they do not fix coverage, self-selection, nonresponse, or misleading wording. Report sample size as one property of the study rather than as a quality grade. The right question is whether the recruitment supports the population named in the conclusion.

Inspect weighting and disclosure

Credible broader estimates explain the sampling frame, selection method, field dates, response process, weighting variables, and limitations.

If results are weighted, state the variables, targets, source of those targets, and how much weights vary. Weighting can adjust measured differences that are represented in the data, but it cannot guarantee correction for unmeasured differences or missing groups. Credible reporting explains both what the adjustment attempted and what uncertainty remains.

Write the narrowest accurate conclusion

If the only known group is participating site visitors, report that group. A useful narrow claim is better than an unsupported population claim.

Draft the conclusion from narrow to broad and stop when the method no longer supports the next step. Respondents to this poll is narrower than all site visitors, and all site visitors is narrower than the general public. Keep field dates and recruitment beside the percentage. A precise narrow statement provides more value than a broad claim that the design cannot defend.

Use a claim ladder

Write several possible claims from narrowest to broadest: respondents to this poll, participating visitors during the field period, all visitors to the site, people interested in the topic, and the general population. Mark the evidence required for each step.

Publish the broadest claim the recruitment and analysis can actually support. This exercise also improves poll design because it reveals when the intended claim requires a sampling frame, outreach, weighting, or expertise that an open poll does not have. If the necessary population cannot be enumerated, state that coverage limitation before discussing response count.

Keep subgroup results especially narrow. A subgroup can have far fewer responses and different recruitment patterns than the total sample. Do not imply a stable difference merely because two displayed percentages are unequal. Report counts, design limits, and whether the subgroup analysis was planned before viewing the results.