Why Evidence Has Layers

Why one study is never the whole answer

When a headline reports that a study found something, the useful question is how much weight it deserves. Health evidence arrives in layers, and each layer answers a different question with a different degree of confidence. We think a reader who understands the layers can judge most health claims without specialist training.

A single study estimates an effect once, in one group of people, under one set of conditions. It may be well run and still be wrong about the wider world. The layers exist to reduce that risk.

The pyramid and its limits

The familiar evidence pyramid places weaker designs at the bottom, then case-control and cohort studies, then randomized trials, with systematic reviews and meta-analyses at the top. A 2016 perspective article in the journal Evidence-Based Medicine called this picture too simplistic at times. Its authors separate the systematic review, which is a process for selecting studies, from the meta-analysis, which is the statistical aggregation. They argue that "study design alone appears to be insufficient on its own as a surrogate for risk of bias", and that a meta-analysis of well-conducted trials cannot be equated with one built from observational studies at higher risk of bias.

How certainty is graded

The GRADE approach, described in the Cochrane Handbook, uses four levels of certainty: high, moderate, low and very low. A body of evidence from randomized trials begins at high. A body from non-randomized studies begins at low. Certainty is then lowered for design flaws, unexplained inconsistency, indirect evidence, imprecise estimates or likely publication bias, usually by one level for each factor. It can be raised for a large effect (a risk ratio above 2 or below 0.5 without plausible confounders), for a dose-response gradient, or when all plausible biases would work against the observed effect.

Randomized trials start high for a reason. The Handbook's chapter on bias reports that trials with inadequate or unclear allocation methods tend to give exaggerated effect estimates, and that missing blinding inflates results most when the outcome is subjective, such as pain. Reporting is often poor: across 20,920 trials, 49% were unclear on random sequence generation and 57% on allocation concealment.

A worked example from trial phases

The US Food and Drug Administration describes the phases of clinical research. Phase 1 involves 20 to 100 people and about 70% of drugs move on. Phase 2 involves up to several hundred people and about 33% move on. Phase 3 involves 300 to 3,000 people and about 25 to 30% move on.

Treating those approximate figures as a chain, the share passing all three is 0.70 x 0.33 x 0.25 = 0.058 at the low end and 0.70 x 0.33 x 0.30 = 0.069 at the high end. That is roughly 6 to 7 in every 100. This is our arithmetic on rounded figures, so it is a rough guide only. It shows why an early result is rarely the final answer.

A separate argument points the same way. A 2005 essay in PLoS Medicine argued that a research finding is less likely to be true when studies in a field are smaller and effect sizes are smaller. The essay is a modelled framework, not a count of failed findings. We cite it as a reason for care, not as a measurement.

What the evidence does not show

A high GRADE rating describes confidence in an estimate for a group. It does not say what will happen to any one person. The pyramid is a guide, not a verdict, and we think a well-run observational study can be more informative than a poor trial. The FDA page is also internally inconsistent on Phase 1, giving 20 to 100 volunteers in one place and 20 to 80 in another, which is a reminder that even official summaries are approximate.

Common mistakes follow from this. Readers treat design labels as quality scores. They assume a small trial is biased when it may only be imprecise. The Handbook notes that a small trial can have low bias but wide confidence intervals. They also accept a relative figure without the absolute one.

Going deeper

The Handbook's chapter on effect measures defines risk as the probability that an outcome occurs, and the risk ratio as the ratio of the risk of an event in the two groups. The risk difference is the difference between the observed risks, and the number needed to treat is obtained from the risk difference. In plain text:

• Risk = people with the event / people in the group
• Risk ratio (RR) = risk in intervention group / risk in comparator group
• Risk difference (RD) = risk in intervention group - risk in comparator group
• Number needed to treat (NNT) = 1 / absolute value of RD

Illustrative, not real data: if 2 in 100 people have an event in one group and 4 in 100 in the other, RR = 0.02 / 0.04 = 0.5, RD = 0.02 - 0.04 = -0.02, and NNT = 1 / 0.02 = 50. A risk ratio of 0.5 sounds large, yet the absolute change is two people in a hundred. When you read a study, look first for the comparator, the absolute numbers, the outcome measured and the certainty rating, then ask which layer of evidence you are actually looking at.

We have linked every source we used. If you find a figure we have misread, we would like to hear about it.

Informational only, not medical advice.