Modern medical science develops at an overwhelming pace. Busy physicians look to top-ranked journals and the randomized clinical trials they publish to learn which of the many new developments merit actually changing how to diagnose and treat patients. But retractions in two of the most storied medical journals during the COVID-19 pandemic have shined a spotlight on the trust the healthcare community places in a fragile process — and how that trust can impact thousands of patients very quickly.

The process is academic publishing and the peer review that undergirds it. Data integrity is certainly an issue, but equally pressing are the nuances of statistical analysis that lie outside clinicians’ realm of expertise.

Multiplicity — what it is and why it matters

One of those nuances — multiplicity, or the tendency of the presence of multiple variables and composite analyses to potentially inflate type I error rates — recently attracted the attention of a team of clinicians and statisticians from top medical institutions in the U.S. and abroad. The team published a study in JAMA Network Open earlier this year examining the prevalence of multiplicity in cardiovascular randomized clinical trials.

“The issue of multiplicity is a serious one, because most clinicians do not possess the background in statistics to scrutinize every detail of reported data,” says the study’s corresponding author, Ankur Kalra, MD, Section Head of Cardiovascular Research at Cleveland Clinic Akron General. “Further compounding the problem is the need to pluck the most relevant findings from the formidable volume of data collected in a clinical trial. We felt it prudent to examine the issue of multiplicity in detail. Nothing less than our trust in the scientific process is at stake.”

The analysis in brief

For their study, Dr. Kalra and colleagues searched the top three cardiovascular journals (Circulation, European Heart Journal and Journal of the American College of Cardiology) and the top three general medicine journals (JAMA, The Lancet and The New England Journal of Medicine) over a three-year period for general trial characteristics and multiplicity error and correction.

As detailed in the full study report, the researchers found that 58.7% of trials had some form of multiplicity analysis. Of these, only 28.3% adjusted for multiplicity. No association was found between the reporting of multiplicity risk assessment and funding source or intervention type. Trials assessing mortality were more likely to have multiplicity risk in the primary analysis compared with those assessing nonmortality outcomes. Smaller trials were more likely to make multiplicity adjustments.

A need for standardization

“Currently, there is no standardized way to report multiplicity,” says study co-author Samir Kapadia, MD, Chair of Cardiovascular Medicine at Cleveland Clinic. “Sometimes commentary on this confounder appears in editorials, or in a paragraph in a study’s discussion section, and sometimes it’s included in the analysis. We think this should change. We’re not saying all clinical trial data that doesn’t account for multiplicity is inaccurate, or that multiplicity analysis is even always necessary, especially among hypothesis-generating studies. But as physician scientists, we must come to some consensus on how and when multiplicity should be reported.”

What should be done

Based on the results of their analysis, the study authors recommend that the following steps be taken, at minimum:

• Require a description of trial protocol and specific analytic plan, including multiplicity adjustment methods or an acknowledgement of the lack thereof, for randomized trials published in medical journals.
• Develop a system for determining which outcomes require multiplicity analysis. For instance, in a trial examining all-cause mortality and hospital readmissions, perhaps different stakes call for different ways of addressing multiplicity.
• Remember that there is an art to science, and that the narrowest restriction of analyses can make us so singularly focused that we entirely miss another important observation from the study. State the intent of the study and its relationship to the presence or lack of multiplicity analyses clearly.

“We must remain ever vigilant and committed to the integrity of our data and methods, not only because patient well-being is at stake, but because, in a climate of skepticism about science, any crack in the foundation of how we produce and acquire medical knowledge bears upon the entire profession,” observes Dr. Kalra.

“For our patients to trust what we say, we must trust what we say,” Dr. Kapadia adds. “Standardizing how we perform and report multiplicity analyses is one simple way to maintain that trust.”