Positive single-center randomized trials and subsequent multicenter randomized trials in critically ill patients: a systematic review

Two investigators independently searched PubMed for all RCTs of any non-surgical intervention influencing unadjusted landmark mortality in critically ill patients (> 48 h after randomization), published in three medical journals (i.e., New England Journal of Medicine, JAMA, and Lancet) from inception to December 31st, 2016. We did not consider sRCTs published after 2017 considering the time lag between the publication of sRCTs and their corresponding mRCTs.

We considered a difference in mortality as statistically significant when present at a specific point (> 48 h after randomization) with simple statistical tests and without adjustment for baseline characteristics. We selected articles published in NEJM, JAMA, or the Lancet, with a randomized controlled trial design in a single-center setting, presenting a statistically significant reduction or increase in unadjusted landmark mortality in critically ill patients. A quasi-randomized or non-randomized methodology, multicentric trials, pediatric populations, and absence of data on mortality were considered exclusion criteria. The full PubMed search strategy is available in Additional file 1.

After identification of eligible sRCTs, two investigators independently searched for mRCTs addressing the same PICO (population, intervention, control, outcome) frameworks, which were published from inception to December 31st, 2022.

The risk of bias of each included sRCT was assessed using the Cochrane risk-of-bias tool for randomized trials version 2 (RoB 2) [6].

Two investigators extracted the following variables in a standardized data collection form: PubMed unique identifier, journal, first author, year of publication, study population, number of patients enrolled, intervention, control, mortality data with statistical significance, and timepoint of mortality assessment. If a subsequent mRCT was identified, we evaluated whether the mortality findings of the mRCT were consistent with those of the sRCT. Furthermore, we explored whether sRCTs were incorporated into international clinical practice guidelines. We further assessed whether and when guidelines stopped citing such RCTs or issued recommendations modified by the mRCTs findings.

First, positive sRCTs with at least one subsequent mRCT were classified into three groups based on the results of mRCTs: significant mortality reduction (positive mRCTs), no significant difference in mortality (neutral mRCTs), and significant mortality increase (negative mRCTs). The proportion of sRCTs within each group was reported accordingly.

Second, we categorized included sRCTs that were cited at least once in international clinical guidelines based on the current guideline recommendations: supporting the intervention shown to have survival benefits in the sRCT, withholding recommendation due to insufficient evidence, and opposing the intervention of interest or excluding the sRCT cited in the preceding version of the guidelines.

To confirm the robustness of our findings, we performed a sensitivity analysis including only recent positive sRCTs published after 2001 to describe the mortality results of subsequent mRCTs and the guideline recommendations regarding the intervention assessed in the included sRCTs.

Furthermore, the following data were summarized: the number of randomized patients (sRCTs and mRCTs), the number of participating centers (mRCTs), the duration between publications of the sRCT and subsequent mRCT, and the duration from the initial citation of the sRCTs in the guidelines to an alteration in recommendation against its use or removal from guidelines. Missing data were not imputed throughout this study. Continuous variables were described as median and interquartile range (IQR). Categorical variables were expressed as number (percentage). We used RStudio Version 2023.06.0+421 (RStudio Team, Boston, United States).

Our systematic review found 19 sRCTs with a statistically significant mortality decrease in critically ill adult patients. Most of these were followed by at least one subsequent mRCT. Survival benefits observed in sRCTs were rarely corroborated by mRCTs, with most mRCTs reporting neutral results on mortality, and one mRCT finding a significant mortality increase with intensive glucose control. Treatment recommendations based on the initial citation of sRCTs with survival benefits were included in international guidelines and typically remained unchanged for a decade before any revisions were made based on subsequent relevant mRCTs.

RCTs in intensive care medicine tend to deliver neutral results in terms of mortality for several reasons including heterogeneity of patient characteristics, underlying practice variation, insufficient power, and likely small treatment effects [3]. This fact poses an important challenge for clinicians because they must perform clinical practice without robust evidence supporting their decisions. As a result, positive trials, namely RCTs reporting statistically significant reductions in mortality attributable to the intervention of interest, look attractive and are often taken up by physicians to change their routine management. Unfortunately, however, such positive RCTs frequently suffer from methodological problems, which can limit the applicability of their findings. Furthermore, single-centric design itself carries many other limitations [2, 3].

One of the major challenges of sRCTs is that, to achieve an effect on mortality in the presence of a small sample size, they must achieve an implausibly large effect size. Single-center trials are typically conducted by advocates of the intervention under investigation [2]. The delivery of such interventions generally requires specialized expertise and dedication, which may not be readily transferable to other centers involved in subsequent large mRCTs. Such discrepancies may limit the feasibility of the interventions, potentially diminishing the magnitude of the treatment effects observed in mRCTs compared to sRCTs. In fact, a meta-epidemiological study evaluated the differences in treatment effects between sRCTs and multicenter RCTs and found that single-center trials showed a statistically significant larger treatment effects than multicenter trials (ratio of odds ratios, 0.73; 95% confidence interval 0.64–0.83) [68]. This finding was confirmed by a systematic review assessing treatment effects on mortality in critically ill settings [69]. By pooling 82 eligible RCTs, this systematic review found that a single-center design resulted in larger treatment effects than a multicenter design (ratio of odds ratios, 0.64; 95% confidence interval 0.47–0.87) [69]. Our selection criterion of sRCTs with significant mortality differences is a unique approach; nonetheless, the present systematic review was consistent with previous work showing that survival benefits in sRCTs were rarely replicated in mRCTs. Moreover, we identified one mRCT demonstrating a significant mortality increase by an intervention (intensive glucose control strategy) [34], which reduced mortality within the context of a previous single-center trial [23].

Nearly half of clinical guidelines that cite sRCTs, recommend the relevant intervention based on their positive results, despite some of these endorsements being subsequently refuted in light of accumulated evidence. In addition, a decade was typically required to amend such recommendations from clinical guidelines. Given the pervasive application of interventions examined in RCTs, these initial recommendations might have played a substantial role in the potential consequences on patient outcomes, healthcare resources, and economic costs. For example, early-goal directed therapy was recommended in the surviving sepsis campaign guidelines in 2004 [56]. However, later, three mRCTs found no benefits in clinically relevant outcomes [42, 45, 49]. Furthermore, economic evaluation using one of the mRCTs revealed that early-goal directed therapy was associated with increased health care costs without improving outcomes [70].

Despite these methodological challenges, positive sRCTs have made changes in clinical practice. The early goal-directed therapy for septic shock is a typical example. The initial sRCT [19] showed survival benefits of this protocolized management, which was not replicated in subsequent mRCTs [42, 45, 49]. However, given the difference in patient severity between the sRCT and subsequent mRCTs (e.g., reduced vs. normal central venous oxygen saturation [ScvO₂]), clinicians now pay more attention to ScvO₂ values than before the sRCT [71]. Furthermore, the lack of multicentric confirmation of survival benefits implies a restricted external validity of sRCTs rather than an indication of them producing false positive results.

As the intensive care community advances the methodology of randomized trial design and execution, there remains a notable lack of evidence demonstrating improved mortality from interventions. These disappointing results have been obtained by using frequentist statistics, where the conclusion is dichotomized to yes or no based on confidence intervals and p values. In contrast, Bayesian analysis provides a probabilistic assessment of the magnitude and direction of true treatment effects, which allows clinicians to augment the interpretation of the trial results. Interestingly, there are several intensive care trials where frequentist statistics denied significant mortality reduction, followed by a Bayesian reanalysis revealing a high probability of survival benefits [72, 73]. Therefore, the integration of Bayesian analysis in intensive care trials may offer a solution to the limitations commonly encountered with frequentist approaches.

This systematic review found mortality reduction was rarely replicated in mRCTs despite the existence of previous positive sRCTs. This implies that there are potential risks when incorporating novel interventions into routine practice based on positive sRCTs without mRCTs confirmation. Importantly, no intervention is free from complications. In addition, new interventions tested in randomized trials often consume more human resources and economic costs. Therefore, management change will inevitably result in complications, increased workload, and costs, all of which did not exist with previous usual care. Given the high likelihood of no mortality difference or even mortality increase in subsequent mRCTs, our findings imply that clinicians should wait for a large-scale trial prior to changing practice or at least be very careful in interpreting the results of positive sRCTs.

This systematic review is the first study to comprehensively identify sRCTs reporting statistically significant reductions in mortality and their corresponding subsequent mRCTs in the field of intensive care medicine. The infrequent replicability of survival benefits in mRCTs corroborated the limited generalizability of sRCTs’ findings. Evaluating the impact of sRCTs on clinical guidelines may be a novel approach, but it yields important insights into the development and interpretation of guidelines.

We acknowledge several limitations. First, given our focus was solely on intensive care sRCTs, our findings may not translate to other medical disciplines. Nevertheless, the generic limitations of sRCTs are universal, regardless of the targeted population or intervention type. As such, sRCTs need to be perceived as hypothesis-generating and clinicians ought to assess the results of sRCTs with a balanced consideration of their strengths and weaknesses. Second, our study included only sRCTs published until 2016, thereby excluding more recent sRCTs. Despite this limitation, our primary objective was to compare the mortality findings of sRCTs with those of subsequent mRCTs, which necessitated an intervening period between them. In addition, the median duration between sRCT and mRCT publication was 8 years, providing justification for our inclusion criteria. Third, we included positive sRCTs published in the three renowned general medical journals, excluding those in intensive care specialty journals (as listed in Table S5: Additional file 1). As a result, the number of eligible studies was relatively small; nonetheless, we employed this strategy to evaluate whether subsequent mRCTs could replicate the survival benefits observed in sRCTs with rigorous methodologies. Given the high standards of the included studies and the concordance of the results with previous literature, it is plausible that our findings could be extrapolated to positive sRCTs reported in specialty journals. Finally, our search was confined to international guidelines to explore sRCTs’ citations. This approach was chosen to ensure the quality of evidence synthesis and generalized perspective.