Compare Evidence From Small Samples
Evaluate small-sample evidence through case selection, measurement quality, effect size, consistency, and how much the decision depends on it.
Grade the evidence
- Check how cases were selected.
- Inspect measurement and missing outcomes.
- Compare effect direction and magnitude.
- Limit the claim to the observed context.
Small does not mean useless
A few well-observed cases can reveal a failure mechanism, while a large noisy sample can answer little. The problem is overstating frequency or generality beyond what the cases support.
Rare harms need different logic
One verified severe failure may justify action even when frequency is uncertain. Consequence and preventability matter alongside count.
Example: five customer interviews
Five interviews cannot estimate market preference, but all reveal the same workflow failure with direct screen evidence. The team uses them to redesign the task, not to claim 100 percent of customers agree.
The decision rule
What to record before acting
Match the claim to what the sample can observe, and keep frequency claims appropriately narrow.
Apply the rule in context with grade source fit and method. Keep the evidence and assumptions with the decision so an outcome review can distinguish reasoning from luck.
Continue within evidence and uncertainty
Leading Indicators for a Decision
Choose leading indicators that appear before the outcome, connect through a credible mechanism, and can trigger a specific decision or correction.
Leading Indicators for a DecisionRobust vs Optimal Decision
Prefer a robust option when it performs acceptably across plausible futures and the apparently optimal option depends on one fragile forecast.
Robust vs Optimal Decision