Introduction
A foundational epistemological principle underpinning evidence-based medicine (EBM) is based on the assumption that the estimates of the effects of health interventions are closer to the “truth” if they are based on higher than on lower quality (certainty) of evidence (CoE).1 If the estimated treatment effects are close to the “true” effects, this would also imply that they would less likely to change as evidence accumulates after new studies are completed. Conversely, because its relation to the “truth” is less certain, this also implies that the estimated effects when evidence is of low quality would more likely change in future research. Research to date indicates that guideline panels are willing to issue stronger recommendations when they deem evidence to be of high quality, thus indirectly affirming this central EBM assumption.2-5
However, whether this indirect assessment of quality of evidence based on guidelines panels’ decision-making is accurate is not known. It is possible that current methods of critical appraisal of CoE do not discriminate well between “true” accurate from inaccurate estimates of treatment effects. That is, the effects of health interventions based on low quality of evidence may turn out to reflect “true effects” by testing in subsequent studies. On the other hand, what was originally deemed as high quality evidence may be undermined by future studies more often than initially expected. Thus, it is not known if low quality evidence is more often revised than high quality evidence. Empirical evidence supporting this foundational principle of EBM is lacking.
The main purpose of this report is to assess if a) low certainty evidence is more often revised than high certainty evidence in subsequent studies, and if b) the magnitude of effect size differs between high and low CoE.