Pilot implementation of short message service for randomisation in a multisite pragmatic factorial clinical trial in Kenya

Abstract

Background

The traditional use of sealed envelopes for randomisation is susceptible to manipulation and the risk of damage to envelopes during shipping and storage. Additionally, the filling and sealing of envelopes are tedious, time-consuming, and error-prone. Other randomisation alternatives such as web-based methods are preferred. However, they are expensive and unsuitable in settings with poor internet infrastructure. Mobile phone-based randomisation using short message service (SMS) potentially offers a low-cost and reliable alternative.

Methods

We developed an SMS-based method for random allocation of treatments. Plain text messaging or an Android app was used to formulate text messages using a fixed syntax consisting of the participant’s unique identifier, trial site, stratum, and the trial name as input parameters. The system verified the input parameters and obtained an allocation from the database before returning a response to the sender. The text response contained the details of the treatment allocation. This was a Study Within A Trial (SWAT) conducted in two sites of a multi-site 3 × 2 factorial clinical trial in Kenya involving two interventions with up to nine possible allocations. SMS randomisation feasibility was assessed by comparing treatment allocations against the master randomisation list for each processed SMS, measuring SMS latency (in seconds), and gathering user feedback via a post-implementation survey.

Results

A total of 218 participants were randomised between the 7th of February 2022 and the 11th of April 2022, out of which 179 were randomised to only one arm while 39 were randomised to both treatment arms. Allocation accuracy was 100%. Median latency was 22 s with the fastest message processed in 10 s and the slowest (non-network delayed) message processed in 2129 s. Four users completed a post-implementation survey.

Conclusions

The pilot study demonstrated that SMS randomisation is easy, user-friendly, fast, accurate, and a feasible alternative randomisation technique.

Background

Randomisation of participants in clinical trials has become the standard method of experimental control aimed at reducing selection bias and eliminating confounding from known and unknown factors [1]. The process of randomisation generally involves two steps: (i) generating an unpredictable sequence of random assignments and (ii) implementing the sequence in a way that conceals the treatment assigned to potential study participants until eligibility is determined [2, 3]. Failure to achieve proper randomisation and allocation concealment may result in biased estimates of treatment effects and potential loss of integrity of trial results [4].

Traditionally the use of sequentially numbered, opaque, sealed envelopes has been regarded as an acceptable method for concealing allocation of interventions in trials. However, this method is now falling out of favour due to its vulnerability to manipulation [5]. Furthermore, sealed envelopes are susceptible to damage during shipping and storage. The process of filling and sealing envelopes is also a time-consuming manual process that is prone to human error, particularly in large complex studies.

In response to the limitations associated with sealed envelopes and recognising inadequate methodological approaches in controlled trials, there is a growing inclination towards the adoption of centrally administered web-based or telephone-based randomisation in large studies. However, implementing these methods is challenging in settings with inadequate communication infrastructure [6] and unreliable internet connectivity.

An alternative approach to randomisation, which is low in cost, auditable, and particularly suited for low- and middle-income countries where access to mobile phone technology has rapidly expanded [7, 8], involves the use of mobile phone-based short messaging service (SMS). SMS is a method of communication that transmits text messages up to 160 characters in length, among mobile devices or from a computer to a mobile device. Kenya is reported to have 98% mobile penetration amongst adults [9].

Bulk messaging enables the synchronous delivery of SMS text messages to a vast number of recipients minimizing delays and overlapping requests. In clinical trials, text messaging has proven effective in reducing missed appointments [10] and has served as a cost-effective intervention for managing patients with chronic illnesses [11,12,13,14].

We developed an SMS-based method for the random allocation of treatments and subsequently undertook a pilot study comparing an SMS-based randomisation platform versus the conventional approach using sealed opaque envelopes. The study was conducted in parallel with a 3 × 2 factorial pragmatic randomised controlled trial of alternative treatments for severe pneumonia among children aged 2–59 months [15].

Our aims were to evaluate the feasibility and accuracy of randomisation using text messaging by estimating the response time of SMS delivery for randomisation requests, assessing the user experience for envelope randomisation and SMS randomisation approaches, correct treatment allocation, and determining allocation sequence concordance for envelope randomisation and SMS randomisation.

Methods

We conducted a prospective two-arm pilot study nested within an actively recruiting randomised controlled trial. This study was conducted in two phases. The development phase (phase 1) involved the design specification of the SMS platform, and initial testing in web-based, text messaging, and Android applications. The implementation phase (phase 2) involved the deployment of the application at two public hospitals in Kenya: Machakos Level 5 and Mama Lucy Kibaki Hospitals selected purposively from a pool of 12 clinical trial sites due to their high participant recruitment rates. A consecutive sampling method was employed at each site to recruit 200 participants.

Phase 1: Development phase

We designed and developed a three-tier SMS-based randomisation system consisting of data, application, and presentation interfaces (Fig. 1). The requirements of the application were derived from standard operating procedures for randomisation in the larger clinical trial. Therefore, the logic was structured to accommodate a multi-step factorial randomisation design involving two interventions with up to nine possible allocations (Fig. 2).

Fig. 1

SMS platform design framework. The data tier stored all the system data, the business logic tier processed all the system transactions, and the presentation tier was the point of interaction between the user and the system

Fig. 2

Factorial allocation of treatments. Three antibiotic treatment arms (crystalline penicillin and gentamicin, ceftriaxone, and intravenous (IV) amoxicillin-clavulanic acid) and two supportive care treatment arms (Nasogastric feeds and IV fluids)

Each message consisted of a predefined ordered syntax comprising the participant’s unique identifier, trial site, stratum, and trial name. Detailed descriptions of the syntax, message scenarios, and expected responses are provided in Tables 1 and 2.

Table 1 SMS formulation syntax of a randomisation request

Table 2 Various SMS formulation request scenarios and their respective expected responses

The application tier verified the input parameters received from the mobile network operator through SMS or hypertext transfer protocol via a Representational State Transfer Application Programming Interface (REST API), obtaining an allocation from the data tier stored on a local database. It then returned a response to the sender through an SMS. The text response contained details of the treatment allocation, participant identifier, and identity of the study staff undertaking randomisation. The system was designed to identify duplicate randomisation attempts using unique patient identifiers (IPNO).

The randomisation application logged all the SMS processed (Table 3). These included invalid text messages, duplicated attempts to randomise, non-authorised requests from users not registered, and successfully processed valid randomisation requests. Valid randomisation and allocations were logged and captured in the administrative dashboard for review during trial monitoring. This log captured allocations for the two clinical trial arms—antibiotic care and supportive care. Antibiotic treatment allocation was the first step of randomisation consisting of three antibiotic regimens: crystalline penicillin and gentamicin, ceftriaxone, and intravenous (IV) amoxicillin-clavulanic acid. A supportive care arm was allocated as the second randomisation step consisting of two treatments: Nasogastric (NG) feeds and IV fluids (Fig. 2).

Table 3 SMS request categories

The administrative dashboard was a central hub for monitoring all transactions and randomisation logs. It stored the randomisation sequence, which was uploaded, during the set up allowed administrators to supervise trial randomisation, manage users, and track recruitment across different sites and strata. A feature phone and a mobile application served as randomisation access points.

The mobile application was developed in Java for Android, and the web-based platform was developed using the hypertext preprocessor scripting language Laravel framework. The platform is integrated with an SMS Application Programming Interface (API) from a local premium rate service provider (PRSP). One local mobile network operator was chosen for piloting due to cost-related estimations. The source code for the SMS dashboard and the mobile application of this project is archived on GitHub [16, 17]. The web-based administrative dashboard is locally hosted on the KEMRI-Wellcome Trust servers following data management procedures outlined in the study protocol. At the time of development of this manuscript, the mobile application had not yet been published on the Google Play Store.

Phase 2: Pilot SMS randomisation

Four clinical trial clinicians carried out the SMS randomisation pilot, with two clinicians stationed at each trial site. All users underwent training on how to use the SMS randomisation prior to piloting. The SMS platform was implemented in two modes: through text messaging on feature phones and smartphones using an Android mobile application. SMS randomisation was conducted alongside the traditional method of using envelopes. Each clinician was provided with a tablet computer with a subscriber identity module card registered to the study. The custom Android application was installed on each of the tablets with each clinician having a separate account with a designated role to randomise participants. All system users were pre-registered in the database.

A study clinician would first screen patients for eligibility and then proceed to randomise them using sealed envelopes (the primary method) and finally repeat the process using text messaging. Randomisation requests were submitted in structured text format, either manually typed in the phone’s default text messaging application or formulated automatically by the Android application. Texts from the Android app included a phone number at the end while manually typed texts did not. From a design standpoint, we would not expect substantial latency variations between the plain text and mobile app requests. Both requests were processed by the same API algorithm. The sole difference was in the user interface for initiating the request: one involved a mobile application, while the other used plain text sent to a specific number code.

Randomisation marked the final step in recruitment before treatment was allocated to a participant. Treatment allocation and administration were based on the envelope concealment method. Patient care was always the priority, ensuring that the study procedures did not delay or interfere with treatment. There was no direct risk to participants from the procedures of this study. If technical issues arose, the clinicians were able to call the user support team at the KEMRI-Wellcome Trust Programme for help. Additionally, weekly review meetings were held to assess progress and address any emerging challenges.

A post-implementation survey was used to evaluate user feedback. We built a user feedback questionnaire into the mobile application, which only became active after the pilot implementation was completed. Each user of the randomisation module completed the questionnaire. The survey explored challenges associated with envelope-based randomization, user preferences for randomization methods (plain text vs. mobile app), understanding of text randomization processes, obstacles encountered during text randomization, time spent on the process, preferred method (plain text or mobile app), and suggestions for improving the text randomization experience.

The study covered the cost of the premium SMS subscription package, ensuring that users did not incur any additional charges.

Feasibility outcomes

The following parameters were used to measure feasibility:

• Latency: Average response time (in seconds) for SMS delivery following each randomisation request.

• Allocation accuracy: Concordance between treatment allocation sequence and the expected randomisation list.

• User feedback: Subjective experience from users comparing envelope and SMS randomisation approaches.

Sample size justification

Pilot studies are not powered like definitive trials [18, 19]. Our sample size was guided by feasibility assessment needs, rather than a formal power calculation for efficacy [18, 20]. The initial target sample size for this pilot study was 200 participants; however, 218 participants were ultimately randomised using the SMS approach. Recruiting beyond the original target allowed us to gather a richer dataset for assessing feasibility outcomes, particularly the latency of the SMS system under varying traffic loads and across the 3 × 2 factorial design allocation possibilities. This approach aligns with recommendations to maximise information gained from pilot studies, even if the sample size exceeds initial targets [18].

Data analysis

The SMS platform logged data for each SMS request made, capturing both the initiation time of the request and the time a response was delivered to the user. This enabled us to calculate a turnaround time, or SMS latency, in seconds for each processed message. We then analysed the data by computing the medians with interquartile ranges (IQRs) for turnaround time in seconds. To better describe the range of SMS requests made, we grouped all requests into six distinct categories with each SMS assigned to a group (Table 3). We only computed the SMS latency for valid randomisation requests (group 5 in Table 3). A valid request was defined by the correct structured syntax with all the input parameters required for randomisation. We calculated the percentage of requests with valid syntax that were successfully processed and resulted in an allocation treatment being delivered to the user, along with the corresponding binomial exact confidence interval (CI).

To determine the validity of an (In-Patient Number) (IPNO), we extracted all IPNOs in each SMS request and compared them against the IPNOs in the clinical trial database. We evaluated the accuracy of treatment allocations by comparing SMS request response for treatment allocation with the master randomisation list for each processed message. Survey responses from all the users were reviewed and summarised.

Results

In the testing and pilot phases of the study, we logged a total of 580 SMSs, which we categorised as shown in Table 4. We noted various types of SMS requests. For 151 (26%) that fell under the invalid non-authorised request category, messages consisted of syntax completely unrelated to randomisation, often missing the keyword randomise. The system reported 22 (4%) requests attempting to randomise participants who were already allocated treatments. One SMS reported under the exhausted sequence category was a test case scenario where the allocation sequence was no longer available for randomisation. Two unregistered users made attempts to randomise participants, while 402 (69.3%; 95% CI 65.4 to 73.0%) requests had valid syntaxes that were processed, and an allocation treatment was delivered to the user as a response.

Table 4 Total SMSes processed during the testing and piloting phases of the study

The SMS latency for the valid successful randomisation processed requests is as shown in Table 5. The median latency was 22 s, with the fastest processed SMS taking just 10 s (IQR 29.75 s). We observed one delayed response, which was eventually delivered 35 min later. It stands out as an outlier as the majority of the SMSs were processed in under 100 s.

Table 5 SMS latency IQR table for valid successful randomisation requests

Between February 2022 and May 2022, 218 participants were successfully randomised in the two participating clinical trial sites using the SMS approach. One hundred seventy-nine participants (82.1%) were randomised to receive an antibiotic treatment alone, while 39 participants (17.9%) were randomised to an antibiotic and a supportive care treatment.

Allocation accuracy was 100% when compared to the allocation sequence. Four clinicians completed a post-pilot survey. From the responses, it took a clinician less than 2 min to compose a randomisation text. Two exclusively used the mobile app for randomisation, while two utilised both feature phones and the app. Generally, the clinicians reported that they found SMS randomisation easy to grasp and use. However, opinions on preference were split; two clinicians favoured envelopes, while two preferred text messaging. The Android application was notably preferred over manual texting for composing the randomisation texts. A recurring challenge was forgetfulness in using text randomisation.

Discussion

Our research provides unique insights as, to the best of our knowledge, it is the first study investigating a Mobile SMS randomisation approach in a low-income setting within a complex randomised controlled clinical trial. Envelope randomisation, a manual and traditional method, is reliant on the integrity of filling, sealing, transporting, and storage of envelopes, highlighting the need for digital alternatives such as SMS. In our pilot study, we found SMS randomisation to be user-friendly, efficient, fast, and accurate. It also addressed significant challenges associated with envelope randomisation, such as the time-consuming process of preparing envelopes and uncertainties related to envelopes being unsealed or damaged. Additionally, it eliminates the needs for paper, printing, and shipping.

Given the high mobile penetration and widespread use of SMS messaging [21] with the low cost of SMS at $0.0078 per unit, the potential of SMS randomisation is evident. SMS processing through a local network recorded a turnaround time of 22 s. This highlights the approach’s practical potential in pragmatic trials, ensuring that there are no delays in service delivery within a busy public routine care hospital setting during SMS randomisation. The introduction of the mobile application that automated SMS formulation made randomisation more efficient, and no internet connectivity was needed. Despite the predominance of feature phones in the Kenyan Market [22], our solution demonstrates versatility, proving that mobile applications and feature phones can be seamlessly integrated and used interchangeably for SMS randomisation. This ensures broad accessibility and efficiency across different device types.

Digital randomisation techniques

As randomisation is a key determinant of the effectiveness of a clinical trial, trialists need to embrace improved and innovative methodologies that include the use of technology where applicable. Trialists are increasingly exploring digital options for conducting randomised controlled trials to mitigate challenges affecting recruitment in clinical trials [23,24,25]. Digitization stands out in ensuring correct and accurate treatment allocation. This not only helps in maintaining the integrity of the clinical trial but also plays a crucial role in minimizing and scrutinizing potential biases in trial outcomes, thereby enhancing the credibility of the results. Clinical trial monitoring becomes more efficient as trial progress can be readily traced in real-time through a randomisation dashboard integrated into the trial data collection process. This feature is particularly beneficial for adaptive clinical trial designs, where the ability to make data-driven decisions in real time is paramount [26].

The use of a numeric short code, which can be unintentionally used by unauthorised mobile subscribers, led to a significant increase in invalid, unauthorised requests, particularly for plain-text randomisation. Unlike mobile app randomisation, which is restricted to registered users and pre-validates SMS values before processing, plain-text messaging lacks this safeguard, making it a less reliable option. Successful use of plain-text messaging relies heavily on user diligence and adequate training, whereas the mobile app provides a guided experience that minimises errors. Additionally, messages sent to a number code cannot be strictly controlled, increasing the risk of accidental use by unauthorised individuals.

Clinical trials in low-resource settings

Mobile-based randomisation can solve a number of clinical trial challenges inherent in low-resource settings such as financial constraints, operational barriers such as remote locations of study sites, and limited human capital [27]. Setting up clinical trials with complex designs can be prohibitively expensive in such settings [28]. This calls for effective methods of conducting trials that deliver credible results while minimizing cost. Our approach, developed using open-source tools, serves as a testament to the feasibility of digitizing clinical trial methods in low-resource settings. Representing marginalised populations in health research and innovation is crucial for addressing the significant disease burden in low-income countries in a fair and equitable manner [27, 29,30,31]. There is an urgent need for investment in solutions that will increase the number of clinical trials conducted in low-income countries [32]. As our approach only targets two components of clinical trials—randomisation and trial monitoring—additional research is required to pilot other low-cost tools that could improve the quality of clinical trials in similar contexts.

Limitations and recommendations

We acknowledge various limitations to our study. The pilot was done in a restricted context with a limited number of users and trial sites. Users were clinicians already involved in the larger randomised controlled trial, which could have influenced their feedback and experience. There may be a learning curve or initial hesitations for naïve. Clinicians admitting to often forgetting to send the text request after opening the envelope also highlights a behavioural aspect that could be addressed in future implementations to ensure consistent use of the system. The application logic was informed by a simple randomisation technique and tested in an urban setting in Kenya setting with local SMS service providers.

This paper does not extend the discussion to implementation in other countries or in rural Kenya where network connectivity may be unstable. Nonetheless, our findings are promising and recommend conducting pilots in various settings, clinical trial designs, and geographical locations.

Treatment allocation accuracy depended on the system design. However, because the pilot study aimed to estimate expected latency rather than meeting a pre-defined latency target, no specific feasibility criteria were defined a priori.

Future iterations of the SMS-based system in studies to assess acceptability, adaptability, integration, and practicality at scale applying theories of implementation science could introduce enhancements that optimise reliability and guarantee integrity.

Training recommendations The success of SMS randomisation heavily relies on training. The training should equip participants with the necessary skills to effectively use the system for randomisation and trial monitoring. The training should cover the following key areas: • SMS structure: Participants should learn how to construct an SMS request according to the project’s specific format • Mobile application usage: Training should focus on navigating the mobile application and its features • Randomisation monitor dashboard: Participants should be trained on how to use the dashboard to monitor the randomisation process To ensure a smooth learning experience, all apps are designed with user-friendly interfaces. This will enable new users to quickly and easily navigate the system Additional considerations for dissemination purposes: • Tailored training: The training should be tailored to the specific needs and technical proficiency of the target audience • Hands-on training: Practical, hands-on exercises should be incorporated to reinforce learning • Training materials: Comprehensive training materials, such as manuals, guides, and presentations, should be developed and distributed • Post-training support: Ongoing support, such as help desks or online forums, should be provided to address questions and troubleshoot issues By following these guidelines, the training can effectively disseminate knowledge and skills to the target audience, ensuring the successful implementation of the SMS randomisation process

Conclusions

The promising results from our pilot indicate that there is potential for wider implementation in large-scale clinical trials. The observed improvements in efficiency, high accuracy, and user acceptance point to the viability of SMS randomisation in clinical research in both low- and high-resource settings. We used open-source tools for the development and testing of the SMS platform, ensuring accessibility for further development and improvements. These lessons from this trial serve as a reference point for future low-cost technology-driven innovations to expand the reach and quality of clinical trials globally.