Advertisement
Preserving trust in research requires vigilance and consensus around statistical nuances
Modern medical science develops at an overwhelming pace. Busy physicians look to top-ranked journals and the randomized clinical trials they publish to learn which of the many new developments merit actually changing how to diagnose and treat patients. But retractions in two of the most storied medical journals during the COVID-19 pandemic have shined a spotlight on the trust the healthcare community places in a fragile process — and how that trust can impact thousands of patients very quickly.
Advertisement
Cleveland Clinic is a non-profit academic medical center. Advertising on our site helps support our mission. We do not endorse non-Cleveland Clinic products or services. Policy
The process is academic publishing and the peer review that undergirds it. Data integrity is certainly an issue, but equally pressing are the nuances of statistical analysis that lie outside clinicians’ realm of expertise.
One of those nuances — multiplicity, or the tendency of the presence of multiple variables and composite analyses to potentially inflate type I error rates — recently attracted the attention of a team of clinicians and statisticians from top medical institutions in the U.S. and abroad. The team published a study in JAMA Network Open earlier this year examining the prevalence of multiplicity in cardiovascular randomized clinical trials.
“The issue of multiplicity is a serious one, because most clinicians do not possess the background in statistics to scrutinize every detail of reported data,” says the study’s corresponding author, Ankur Kalra, MD, Section Head of Cardiovascular Research at Cleveland Clinic Akron General. “Further compounding the problem is the need to pluck the most relevant findings from the formidable volume of data collected in a clinical trial. We felt it prudent to examine the issue of multiplicity in detail. Nothing less than our trust in the scientific process is at stake.”
For their study, Dr. Kalra and colleagues searched the top three cardiovascular journals (Circulation, European Heart Journal and Journal of the American College of Cardiology) and the top three general medicine journals (JAMA, The Lancet and The New England Journal of Medicine) over a three-year period for general trial characteristics and multiplicity error and correction.
Advertisement
As detailed in the full study report, the researchers found that 58.7% of trials had some form of multiplicity analysis. Of these, only 28.3% adjusted for multiplicity. No association was found between the reporting of multiplicity risk assessment and funding source or intervention type. Trials assessing mortality were more likely to have multiplicity risk in the primary analysis compared with those assessing nonmortality outcomes. Smaller trials were more likely to make multiplicity adjustments.
“Currently, there is no standardized way to report multiplicity,” says study co-author Samir Kapadia, MD, Chair of Cardiovascular Medicine at Cleveland Clinic. “Sometimes commentary on this confounder appears in editorials, or in a paragraph in a study’s discussion section, and sometimes it’s included in the analysis. We think this should change. We’re not saying all clinical trial data that doesn’t account for multiplicity is inaccurate, or that multiplicity analysis is even always necessary, especially among hypothesis-generating studies. But as physician scientists, we must come to some consensus on how and when multiplicity should be reported.”
Based on the results of their analysis, the study authors recommend that the following steps be taken, at minimum:
Advertisement
“We must remain ever vigilant and committed to the integrity of our data and methods, not only because patient well-being is at stake, but because, in a climate of skepticism about science, any crack in the foundation of how we produce and acquire medical knowledge bears upon the entire profession,” observes Dr. Kalra.
“For our patients to trust what we say, we must trust what we say,” Dr. Kapadia adds. “Standardizing how we perform and report multiplicity analyses is one simple way to maintain that trust.”
Advertisement
Advertisement
How our first century has impacted cardiovascular practice
Review offers comprehensive assessment of the landscape for wearables and more
Cardiac surgeon Patrick Vargo, MD, reflects on his first year as Cleveland Clinic staff
Improved risk prediction for patients is at the heart of Dr. Aaron Weiss’ research interests
Centralization would likely bring better outcomes, experts say, but may not be feasible
Dr. Daniel Burns on mentorship, robotic valve surgery, statistics and more
JACC review makes the case and outlines how to ensure oversight
Editorial lays out best practices from three Cleveland Clinic surgeons