Polishing the Gold Standard: Reporting Multiplicity in Randomized Clinical Trials

Preserving trust in research requires vigilance and consensus around statistical nuances

20-HVI-1965773-CQD-Multiplicity

Modern medical science develops at an overwhelming pace. Busy physicians look to top-ranked journals and the randomized clinical trials they publish to learn which of the many new developments merit actually changing how to diagnose and treat patients. But retractions in two of the most storied medical journals during the COVID-19 pandemic have shined a spotlight on the trust the healthcare community places in a fragile process — and how that trust can impact thousands of patients very quickly.

Advertisement

Cleveland Clinic is a non-profit academic medical center. Advertising on our site helps support our mission. We do not endorse non-Cleveland Clinic products or services. Policy

The process is academic publishing and the peer review that undergirds it. Data integrity is certainly an issue, but equally pressing are the nuances of statistical analysis that lie outside clinicians’ realm of expertise.

Multiplicity — what it is and why it matters

One of those nuances — multiplicity, or the tendency of the presence of multiple variables and composite analyses to potentially inflate type I error rates — recently attracted the attention of a team of clinicians and statisticians from top medical institutions in the U.S. and abroad. The team published a study in JAMA Network Open earlier this year examining the prevalence of multiplicity in cardiovascular randomized clinical trials.

“The issue of multiplicity is a serious one, because most clinicians do not possess the background in statistics to scrutinize every detail of reported data,” says the study’s corresponding author, Ankur Kalra, MD, Section Head of Cardiovascular Research at Cleveland Clinic Akron General. “Further compounding the problem is the need to pluck the most relevant findings from the formidable volume of data collected in a clinical trial. We felt it prudent to examine the issue of multiplicity in detail. Nothing less than our trust in the scientific process is at stake.”

The analysis in brief

For their study, Dr. Kalra and colleagues searched the top three cardiovascular journals (Circulation, European Heart Journal and Journal of the American College of Cardiology) and the top three general medicine journals (JAMA, The Lancet and The New England Journal of Medicine) over a three-year period for general trial characteristics and multiplicity error and correction.

Advertisement

As detailed in the full study report, the researchers found that 58.7% of trials had some form of multiplicity analysis. Of these, only 28.3% adjusted for multiplicity. No association was found between the reporting of multiplicity risk assessment and funding source or intervention type. Trials assessing mortality were more likely to have multiplicity risk in the primary analysis compared with those assessing nonmortality outcomes. Smaller trials were more likely to make multiplicity adjustments.

A need for standardization

“Currently, there is no standardized way to report multiplicity,” says study co-author Samir Kapadia, MD, Chair of Cardiovascular Medicine at Cleveland Clinic. “Sometimes commentary on this confounder appears in editorials, or in a paragraph in a study’s discussion section, and sometimes it’s included in the analysis. We think this should change. We’re not saying all clinical trial data that doesn’t account for multiplicity is inaccurate, or that multiplicity analysis is even always necessary, especially among hypothesis-generating studies. But as physician scientists, we must come to some consensus on how and when multiplicity should be reported.”

What should be done

Based on the results of their analysis, the study authors recommend that the following steps be taken, at minimum:

  • Require a description of trial protocol and specific analytic plan, including multiplicity adjustment methods or an acknowledgement of the lack thereof, for randomized trials published in medical journals.
  • Develop a system for determining which outcomes require multiplicity analysis. For instance, in a trial examining all-cause mortality and hospital readmissions, perhaps different stakes call for different ways of addressing multiplicity.
  • Remember that there is an art to science, and that the narrowest restriction of analyses can make us so singularly focused that we entirely miss another important observation from the study. State the intent of the study and its relationship to the presence or lack of multiplicity analyses clearly.

“We must remain ever vigilant and committed to the integrity of our data and methods, not only because patient well-being is at stake, but because, in a climate of skepticism about science, any crack in the foundation of how we produce and acquire medical knowledge bears upon the entire profession,” observes Dr. Kalra.

“For our patients to trust what we say, we must trust what we say,” Dr. Kapadia adds. “Standardizing how we perform and report multiplicity analyses is one simple way to maintain that trust.”

Advertisement

Related Articles

20-HVI-2013168-digital-electrophysiology-650×450
Digital Health in Electrophysiology and Beyond: The Potential and the Challenges

Review offers comprehensive assessment of the landscape for wearables and more

20-HVI-1915481-CQD-Patrick-Vargo-3
New Staff Surgeon Explains His Affinity for the Aorta and Why He Stayed on After Residency

Cardiac surgeon Patrick Vargo, MD, reflects on his first year as Cleveland Clinic staff

20-HVI-1850425-Dr-Weiss-cardiac-surgery-650×450
Early-Career Cardiac Surgeon Finds a Place to Pair Patient Experience With Research Innovation

Improved risk prediction for patients is at the heart of Dr. Aaron Weiss’ research interests

20-HVI-1880566-Karamlou-CQD
Should Cardiothoracic Surgery Be Regionalized in the U.S.?

Centralization would likely bring better outcomes, experts say, but may not be feasible

20-HVI-1880836-Daniel-Burns-650×450
What Drew One Young Cardiothoracic Surgeon to Cleveland Clinic

Dr. Daniel Burns on mentorship, robotic valve surgery, statistics and more

20-HRT-057-Commandment-MV-CQD
For Success in Mitral Valve Repair, Follow These 10 Commandments

Editorial lays out best practices from three Cleveland Clinic surgeons

Ad