Design, analysis, and reporting of pilot and feasibility trials in anesthesiology: a methodological study

Abstract

Background

Pilot and feasibility studies are effective tools for assessing the feasibility of performing larger-scale studies. These are particularly useful in anesthesiology, where the research overlaps with several other medical and surgical fields. The objective of this meta-epidemiological study is to assess the design and methodology of pilot and feasibility randomized controlled trials (RCTs) in anesthesiology.

Methods

We searched for pilot and feasibility RCTs in anesthesiology indexed in PubMed during a 5-year span between January 1, 2018, and December 31, 2022. We extracted bibliographic information, field of study, type of intervention, trial duration, trial design, use of qualitative data, use of progression criteria, whether the primary objective and primary outcome were related to feasibility, reported feasibility outcomes, and sample size justification. We conducted logistic regression to determine the factors associated with using progression criteria, having primary feasibility outcomes, and using feasibility outcomes to justify the sample size. We controlled for publication year, journal impact factor, source of funding, intervention type, and region.

Results

Our search retrieved 3015 trials, of which 248 were ultimately included and analyzed. Less than a third of studies stated feasibility as the primary objective (n = 77, 31.0%). Feasibility was a primary outcome in 46 (18.6%) studies, progression criteria were used in 27 (10.9%) studies, a sample size justification was listed in 134 (54.0%) studies, and 24 (9.7%) studies used qualitative data. We did not find any statistically significant association between progression criteria and any of the selected variables. Recently published trials had higher odds of having primary feasibility outcomes (odds ratio [OR] 1.39; 95% CI 1.06–1.83). Studies of pharmacological interventions had lower odds primary feasibility outcomes (OR 0.41; 95% CI 0.19–0.90). Recent studies also had higher odds of having a sample size justification based on a feasibility outcome rather than a clinical outcome or similar studies (OR 1.51; 95% CI 1.06–2.15).

Conclusions

More recently published pilot RCTs were significantly associated with having a primary feasibility outcome and determining sample size based on feasibility, while pharmacological studies were significantly associated with less reporting of primary feasibility outcomes. Future research addressing the factors limiting adherence to current guidelines is warranted.

Introduction

The process of conducting primary research can be time-consuming, costly, and resource-intensive. Meticulous planning is often required to ensure the reliability of the methodology, reduce potential bias, and optimize generalizability [1]. Pilot and feasibility studies have emerged as practical tools critical to ensuring the methodological rigor of large-scale research projects. The terms “pilot” and “feasibility” have been used interchangeably even though there is an emerging consensus that pilot studies are a subset of feasibility studies [2, 3]. These smaller-scale studies simulate the methodology and procedures of the proposed definitive study, aiming to assess the feasibility, which can later be adapted to a larger scale. These studies also allow for a setting to find potential associations between variables that researchers may decide to pursue in the larger subsequent study [4]. The value of pilot studies lies in their ability to identify and help mitigate potential logistical issues that would jeopardize the integrity of the larger study [4]. Additionally, pilot studies play a crucial role in reducing the risk of committing resources to trials that have the potential to fail, thereby significantly reducing overall research waste and promoting responsible research practices [5]. Despite their clear value, pilot studies remain underutilized in medical research [6].

Pilot studies are particularly effective at strengthening research in medical specialties like anesthesiology, which frequently forms complex interactions with other medical and surgical specialties. The design and conduct of pilot and feasibility trials provide a unique opportunity to fine-tune research protocols, ensuring that research questions are answered in the interdisciplinary context [7].

There is substantial scholarly activity analyzing the reporting quality of RCTs and RCT abstracts in anesthesiology literature, but there has been no attempt to evaluate pilot and feasibility RCTs [8,9,10]. A cross-sectional study of the reporting quality of pilot and feasibility trials in high-impact anesthesia journals was performed, but it assessed articles from only five journals and extracted a limited number of factors relevant to the methodological quality of studies [11]. This prompts the need for further investigation of more recent studies published in a broader variety of journals while also looking at additional outcomes including bibliographic data, fields of anesthesiology relevant to each trial, use of qualitative data, use of progression criteria, and inclusion of sample size justifications. Progression criteria are a recent development in research methodology. They are pre-specified quantitative thresholds that inform the researchers’ decision to progress to a larger, more definitive trial, allowing for the evaluation of successful or unsuccessful feasibility [12, 13].

Considering the heterogeneous nature of pilot and feasibility, it will also be important to understand the various definitions and methods of reporting feasibility versus clinical outcomes across the trials [14, 15]. This study aims to provide a clearer understanding of how pilot and feasibility trials can be better utilized and standardized within anesthesiology research by systematically analyzing the available studies.

Methods

Database search

We conducted a methodological review of anesthesiology pilot and feasibility RCTs published during a 5-year span between January 1, 2018, and December 31, 2022, in journals indexed in PubMed. The search strategy included terms related to pilot and feasibility studies and randomized trials. The specific keywords included in the search are outlined in Appendix 1.

Data collection

Title and abstract screening was conducted in Rayyan by two reviewers (T. A. J. and A. E. O.) [16]. Discrepancies were resolved by discussion. After importing the remaining references into DistillerSR, four reviewers (T. A. J., M. A., A. E. O., E. I.) completed full-text screening [17]. For studies to meet eligibility, they had to be as follows: (1) a study in anesthesiology (encompassing anesthesia, surgery, pain management, intensive care, emergency medicine), (2) a clinical study, (3) a pilot or feasibility RCT, (4) published between 2018 and 2022, and (5) published in English. Data extraction followed, done by the same four reviewers. During both the full-text screening and data extraction stages, each reference was reviewed by two independent reviewers. T. A. J. resolved any discrepancies in full-text screening and data extraction. The following data were extracted: bibliographic information (author, year, journal, country, country income level, WHO region, source of funding), field of study, type of intervention (pharmacological or non-pharmacological), trial duration, trial design, use of qualitative data, use of progression criteria, whether the primary objective and primary outcome were related to feasibility, reported feasibility outcomes, and sample size justification.

Data analysis

Counts and percentages were reported for categorical variables, while median and quartiles were calculated for continuous variables. We used logistic regression to examine the effects of publication year, journal impact factor, source of funding (private or public, industry, no funding reported), type of intervention (pharmaceutical or non-pharmaceutical), and WHO region (Americas, Eastern Mediterranean, Europe, South-East Asia, Western Pacific) on three important characteristics of feasibility studies: using progression criteria, having a primary feasibility outcome, and justifying sample size based on feasibility outcomes. The model was assessed using Akaike’s information criterion (AIC).

Results

Search results

Our search retrieved 3015 articles, of which 2709 were excluded during the title and abstract screening process. The remaining 306 articles then went through full-text screening, with 35 articles excluded for not being related to anesthesiology, 21 articles excluded because they were not RCTs, and 2 articles excluded because they were not clinical studies. A total of 248 articles were included for data extraction. The screening process is shown in Fig. 1.

Fig. 1

Flow diagram for study selection

Study characteristics

Of the 248 included studies, the greatest number of trials (n = 61) were published in 2020. One-hundred eighty-one (73.0%) of these studies were conducted in high-income countries. The greatest number of studies was conducted in the Americas (n = 83, 33.5%), followed by Europe (n = 77, 31.0%), the Western Pacific (n = 52, 21.0%), South-East Asia (n = 20, 8.1%), and the Eastern Mediterranean (n = 15, 6.0%). One study was multicenter and spanned several WHO regions. There were 105 (42.3%) pharmacological studies. The median duration of each trial was 11 months. Additional study characteristics are listed in Table 1.

Table 1 Study characteristics (n = 248)

Pilot and feasibility characteristics

Regarding nomenclature, 196 (79.0%) studies included the words “pilot” or “feasibility” in the title. Fourteen (5.6%) studies indicated their pilot or feasibility study status using similar words, such as “exploratory,” “preliminary,” or “proof of concept.” The remaining 38 (15.3%) studies did not make any reference to being a pilot or feasibility study in the title.

Studies that primarily assessed feasibility tended to outline it clearly in the abstract. Seventy-seven (31.0%) of the included articles used feasibility as the primary objective. The remaining 177 (69.0%) studies listed primary objectives not related to feasibility, including clinical outcomes and efficacy metrics.

A feasibility outcome was used as the primary outcome in 46 (18.6%) studies. Across the included trials, 14 different feasibility outcomes were reported. Of these primary feasibility outcomes, the most commonly reported were enrolment (n = 49, 19.8%), compliance (n = 30, 12.1%), data completion (n = 25, 10.1%), and retention (n = 23, 9.3%). Table 2 provides details of study designs and methodological outcomes across included trials.

Table 2 Study design and methodological outcomes (n = 248)

A sample size justification was provided in 130 (52.4%) studies. The most common sample size justification was based on a clinical outcome (n = 91, 36.7%), in which the study would recruit enough participants to meet a predetermined statistical power and type-1 error. Forty-six (18.6%) studies referenced literature and similar pilot or feasibility studies to determine an appropriate sample size. Twenty-four (9.7%) studies made a sample size estimation for the purpose of providing sufficient statistical precision for feasibility objectives. Four (1.6%) studies used other justifications for sample size, such as statistician recommendations. Of the included studies, 27 (10.9%) used progression criteria, and 24 (9.7%) incorporated qualitative data.

Multivariable analyses

None of the factors were associated with using progression criteria. Recently published trials had higher odds of having primary feasibility outcomes (odds ratio [OR] 1.39; 95% CI 1.06–1.83), and studies with pharmacological interventions had lower odds of having primary feasibility outcomes (OR 0.41; 95% CI 0.19–0.90). Recent studies had a higher odds of having a sample size based on feasibility (OR 1.51; 95% CI 1.06–2.15). Table 3 provides the results of multivariable regression analysis.

Table 3 Factors associated with progression criteria usage, feasibility as the primary outcome, and feasibility outcome for sample size justification

Discussion

In our review, we found that more recently published studies were significantly more likely to report feasibility as the primary outcome and significantly more likely to have a sample size justification based on feasibility outcomes. These findings may indicate that the recent emphasis on feasibility in pilot studies is being increasingly adopted as more specific feasibility indicators, and guidelines are continuously created [18].

We also found that pharmacological pilot and feasibility studies were significantly less likely to report feasibility as the primary outcome. We are unsure of the exact reason for this finding, but one potential interpretation of this result is that the pharmacological pilot RCTs tended to use primary clinical outcomes rather than feasibility outcomes in order to match the design of the definitive RCT. Another potential reason for this finding is that these studies may have prioritized analyzing the clinical effect of the pharmacological intervention itself, which would be more readily assessed using clinical outcomes rather than feasibility outcomes. Studies in this scenario could consider evaluating the feasibility of the study and the pharmacology concurrently by having both a primary feasibility outcome and a secondary clinical outcome.

One limitation of this study is that we only extracted data from articles indexed in PubMed. Including articles indexed in other databases such as Embase would have expanded the scope of our search and increased the generalizability of the results. Despite including studies from different WHO regions, the exclusion of articles not published in English may have introduced bias. Another limitation is that the implementation of progression criteria is a relatively recent development, meaning that guidelines on their recommended usage have not yet been well established [19]. Further, we report on a 5-year period ending in 2022, precluding us from making more current inferences.

This study does come with strengths. Firstly, our assessment of several methodological outcomes allowed us to more accurately determine the quality of pilot and feasibility RCTs in anesthesiology and gave us a more holistic picture of which factors held significant statistical associations. Additionally, our inclusion criteria were able to capture the various medical and surgical subspecialties which fall under the larger specialty of anesthesiology.

Our findings align with those of similar studies. In a meta-epidemiological study of the reporting of progression criteria in pilot trial protocols, it was found that more recently published protocols were significantly associated with higher odds of reporting progression criteria [12]. This strongly aligns with our findings of more recently published pilot RCTs being significantly associated with reporting feasibility as the primary outcome and having a sample size justification based on feasibility outcomes. These findings support the idea that the reporting quality of feasibility trials is increasing as time goes on.

In a cross-sectional study of the reporting quality of pilot and feasibility trials in the five highest-impact anesthesia journals, it was found that significantly poor reporting was associated with a lack of trial registration, not identifying the trial as a pilot, and using a clinical hypothesis as the primary objective [11]. We looked at factors associated with specific methodological outcomes rather than poor reporting quality in general, so our significant statistical associations varied. But both studies reported less than 40% of included articles reporting key methodological outcomes, such as stating feasibility as the primary objective and primary outcome.

Recent studies are significantly better in terms of feasibility reporting, and increased research on the topic of feasibility trials should allow the reporting quality to continue to improve over time. The CONSORT 2010 extension to pilot and feasibility RCTs is a comprehensive guideline to follow, and our findings prompt further research to explore potential barriers preventing researchers from utilizing it to guide their methodology [20].

Conclusion

Feasibility RCTs published more recently were significantly associated with reporting feasibility as the primary outcome and having a sample size justification based on feasibility, while pharmacological studies were significantly less likely to report feasibility as the primary outcome. Future research should focus on the improved implementation of current feasibility trial guidelines and the barriers which prevent researchers from adhering to them.