Evaluating the Impact of a Game (Inner Dragon) on User Engagement Within a Leading Smartphone App for Smoking Cessation: Randomized Controlled Trial

Introduction

Background

Cigarette smoking remains a leading cause of preventable death and illness in the United States [1]. Although various clinic- and phone-based smoking cessation treatments are effective, the most effective cessation treatment, which combines counseling with medication, is used by fewer than 5% of individuals attempting to quit smoking [1-3]. Most individuals attempting to quit smoking do so without any assistance, resulting in success rates of <5% [4].

The advent of smartphone-delivered smoking cessation support offers a compelling alternative to traditional clinical treatments, given its potential reach and convenience. Smoking cessation apps, in particular, have gained popularity due to their low cost and just-in-time support.

Research shows that English-language smoking cessation apps have been downloaded 33 million times as of 2020 [5]. The widespread ownership of smartphones has contributed to the extensive reach of cessation apps. As of 2023, 90% of US adults, including 84% of Black adults, 91% of Hispanic adults, 79% of adults in low-income households (<US $30,000 per year), and 89% of adults aged between 50 and 64 years, owned a smartphone [6]. Consequently, smartphone-based interventions hold promise to reduce the substantial burden of tobacco-related disease and mortality.

While smoking cessation apps offer the advantage of providing timely assistance, only a small proportion of the numerous apps available in app stores have undergone rigorous testing. A systematic review conducted up until 2019 indicated that 11 smoking cessation apps had been subjected to randomized controlled trials (RCTs), although only 3 were beyond the pilot or feasibility stage [7]. Since then, there have been additional RCTs, although the literature is still emerging. Early-stage evaluations and the full-scale trials have indicated that smartphone cessation apps tend to yield intention-to-treat abstinence rates from 12% to 32% (usually at 1-2 months), with substantial variation due to differences in measures, the inclusion of bioverification, and other design factors [8-20]. However, the translation of these early-efficacy rates into real-world effectiveness remains largely uncertain, particularly regarding app features that enhance user engagement and contribute to sustained abstinence [21].

A major contributor to the modest efficacy rates for many smoking cessation apps is low user engagement and retention [22-24]. Systematic data for smartphone cessation apps are lacking, although evidence suggests a pattern of low retention. Three-quarters of mobile health apps are opened <10 times [25].

To address this gap, researchers have explored the use of serious games and gamification techniques as potential strategies to enhance user engagement and motivation [26-29]. Serious games have a primary purpose other than entertainment, such as health promotion. Gamification is a related motivational tool that uses nonmonetary rewards to make nongame activities fun or challenging. By increasing user engagement, games and gamification may increase exposure and adherence to a smoking cessation program and, in turn, increase the likelihood of quitting [30]. Games may have particular appeal to individuals who smoke, three-quarters of whom play video games, according to 1 survey [31]. Many smokers also express belief in the motivating potential of game-based approaches for smoking cessation [31,32].

Several stand-alone smartphone-based games for smoking cessation, such as Cigbreak, Crush the Crave, Inspired, QuitIT, Quittr, and Tobbstop, have been developed [12,19,20,33-39]. A few of these games have also been tested in RCTs. While users generally reported satisfaction with the gamified apps, most studies have not observed significant increases in smoking abstinence [12,19,20,38]. The variability in user engagement and retention across these apps further complicates the assessment of their overall effectiveness, with each study adopting different measurement approaches.

This Study

We conducted an RCT to test the efficacy of a novel game intervention called Inner Dragon. Whereas prior games have been stand-alone products; Inner Dragon was integrated into the Smoke Free app (23 Limited), providing a unique opportunity to estimate the incremental benefit of embedding a game in a high-quality smoking cessation app. Smoke Free has been found to increase continuous abstinence at 6 months (12.7% intervention vs 7% comparator) on a per-protocol basis in a pragmatic RCT of 3000 participants [15]. Smoke Free is also one of the most downloaded smoking cessation apps in the Apple (Apple Inc) and Android (Google LLC) stores, with more than 800,000 downloads per year and 7 million downloads to date [14,40].

Inner Dragon is a multifaceted game iteratively developed by our team that uses virtual pet retention mechanics with some social features and customization options to promote user engagement and retention [41]. In the Inner Dragon game, users care for an evolving, customizable pet dragon, whose growth reflects the user’s own progress toward quitting smoking. Points are provided for engaging with game elements and nongame educational content. The game includes elements aimed at coping with cravings and offers asynchronous interaction with other users’ dragons. In a single-arm feasibility trial, Inner Dragon was found to have high user satisfaction and generally high user engagement [41]. This study aimed to assess whether the novel Inner Dragon game integration increased user engagement among users of a leading smartphone app for smoking cessation.

Methods

Trial Design

We conducted a parallel-group RCT with equal allocation to 2 groups. The treated group received free access to the Smoke Free app with the integrated Inner Dragon game. The control group received free access to the Smoke Free app without the game embedded in the app. The study protocol is provided in the supporting information with a CONSORT (Consolidated Standards of Reporting Trials) guideline checklist (Multimedia Appendix 1 [6,42-62] and Multimedia Appendix 2) [63]. The protocol was preregistered at ClinicalTrials.gov (NCT05227027). No changes to the prespecified trial design and methods were made after trial commencement. Multimedia Appendix 1 provides the study protocol.

Recruitment and Participants

Individuals were recruited from the US-based population of general users of the Smoke Free smartphone app who downloaded the app from the Apple or Android app store. Users were invited to participate in the trial during the initial onboarding screens within the Smoke Free app via an on-screen recruitment message, “Interested in a research study to test the latest version of Smoke Free?” If a user clicked “Yes, learn more,” they were taken to a screening questionnaire to assess eligibility and a consent form in Qualtrics (Qualtrics International Inc).

Inclusion criteria were individuals aged ≥18 years living in the United States who currently smoked at least 1 cigarette per day; had downloaded the Smoke Free app; planned to quit smoking within 7 days; and were able to speak, read, and write in English. A quota of at least 100 individuals aged ≥50 years was programmed into Qualtrics to ensure the study was representative with respect to age and to enable an assessment of whether older individuals, who have a higher smoking prevalence and may be less proficient with technology or less drawn to games, had a differential response to the intervention [31,64,65]. There were no exclusion criteria.

Randomization and Masking

Eligible, consenting participants were randomized within strata with equal allocation to one of two groups: (1) a control group that received free access to the original Smoke Free app or (2) a treated group that received free access to the Smoke Free app with the integrated Inner Dragon game. Allocation, concealed from investigators, was performed through the Qualtrics randomizer and transmitted to the Smoke Free database using the Qualtrics application programming interface. Participants were then automatically given access to the assigned intervention within the Smoke Free app. Randomization was stratified by age (<50 and ≥50 years) and smoking intensity (<5 and ≥5 cigarettes per day, on average), to facilitate the assessment of effect modification.

The assessor of the salivary cotinine test was blinded to the participants’ group assignment; other aspects of the trial were not blinded.

Procedures

Overview

As part of the screening questionnaire, participants selected an initial quit date within the next 7 days. Participants were invited to use the Smoke Free app in whatever way they liked for 56 days after their planned quit date. Participants were asked via email to complete a baseline questionnaire in Qualtrics after consenting to participate in the study. The baseline questionnaire included additional questions about the person’s demographic and smoking history. A follow-up questionnaire in Qualtrics was performed 8 study weeks (56 days) after the person’s initial quit date (56-63 days after screening and consent depending on each participant’s selected quit date). An invitation to complete the follow-up questionnaire was sent by SMS text messaging. Those who reported having abstained for 7 days in the follow-up questionnaire were asked for a mailing address within the follow-up questionnaire. We then mailed them a salivary cotinine test kit and written and pictorial test instructions via 2-day mail. Following completion of the follow-up questionnaire, an initial SMS text message explained that the saliva test would be shipped shortly, and a second message sent 2 days later provided a Qualtrics link with which participants were asked to upload photos of the cotinine test results. Participants received 3 reminders via SMS text messaging and 2 via email to complete the follow-up questionnaire, and they received 3 reminders via SMS text messaging to complete the saliva testing. In-app reminders were disabled so as not to conflict with the study procedures.

Control Group Intervention

Participants in the control group received free access to an educational intervention consisting of the original Smoke Free app, which has previous per-protocol trial evidence of continuous abstinence at 6 months [15]. The app leverages behavior change techniques that are effective in face-to-face behavioral support programs [66]. “Core” features of the original Smoke Free app include the following: (1) a calculator that tracks the total amount of money saved by not smoking; (2) a calendar that tracks the time elapsed since the user quit smoking; (3) a scoreboard that awards badges to users for not smoking; (4) progress indicators that inform users about health improvements that the user can expect because they started their quit attempt; (5) daily missions that assign evidence-based tasks to help users avoid and resist urges to smoke (eg, noting situations when the person smokes and plans for handling those situations without smoking); (6) a diary and craving log that track the frequency, strength, and location of cravings to smoke; and (7) a text-based chatbot that delivers quitting guidance in a friendly, conversational tone. In previous randomized trials, a full version of the app with a chatbot was shown to increase user engagement compared with a reduced version of the app [67], and the daily missions were shown to increase user retention and self-reported smoking abstinence at 3 months [14]. The current core version of the Smoke Free app does not contain any additional gamification features.

Treated Group Intervention

Participants in the treated group received free access to core features of the original Smoke Free app as well as the integrated Inner Dragon game module. Inner Dragon uses traditional virtual pet retention with some social features and many options for customization and personalization. Virtual pets have been popular with consumers because they can foster bonding and companionship [68].

More specifically, Inner Dragon includes several game mechanics designed to increase user engagement and retention. First, the user maintains a pet dragon that hatches on the user’s quit day and evolves every 7 days to unlock new attributes and powers. The dragon acts as a virtual avatar that represents the user’s progress. Second, the user earns experience points by engaging in selected activities, including those in the game (eg, playing a minigame or feeding the dragon) and nongame features in the original Smoke Free app (eg, completing a mission or logging a diary entry). Rewards for using core app features in the original app were designed to improve adherence to the smoking cessation program. The experience points unlock gifts and cosmetics for the dragon, directly rewarding frequent and consistent use of the app. The user can customize their dragon (eg, wing shape or clothing accessories) throughout the quit experience by steadily unlocking features. Third, the Inner Dragon home screen has “care meters” that users must work on to keep from falling very low: calmness, nutrition, hygiene, and energy. Engaging with the dragon in various ways increases the readings on meters. For example, petting the dragon increases calmness, and feeding the dragon increases nutrition. Caring for and interacting with a virtual pet through the care meters can foster a bond with the pet and motivate users to return to the game regularly [68]. Fourth, the game provides tools for the user to better cope with cravings: a dragon-led breathing exercise to provide calmness and a memory minigame as a distraction. Fifth, a user can asynchronously interact with other users’ dragons in a “dragon park” by (1) viewing their profile and progress and (2) sending and receiving motivational messages and emojis from a preset menu. We hypothesize that these game mechanics increase users’ engagement with the app and, subsequently, their chance of quitting successfully.

Inner Dragon’s game design was informed by principles from the fields of psychology and behavioral economics [41]. The avatar provided salient, visual feedback with endogenous value (tied closely to the user’s motivation to quit) that may sustain and enhance motivation to quit. Self-determination theory predicts that this type of feedback is highly intrinsically motivating [69]. Furthermore, the user may identify with the digital pet as an avatar, and this may cultivate a digital therapeutic alliance with the game and app, for example, by creating a bond with the dragon and increasing the user’s confidence to succeed [42,70]. The use of frequent, salient in-game rewards was designed to counter the behavioral economic constructs of present bias and inattention to app use [71,72]. The design further included evidence-based practices from contingency management, such as the use of escalating in-game rewards for abstinence, with a reset point for lapses and sustained abstinence (by harnessing regret and loss aversion) [73]. The use of surprise gifts provided a variable reward structure designed to boost engagement and novelty [74-76]. The asynchronous interactions with other users in the dragon park provided opportunities for limited social support and social comparisons that may motivate the user to exert more effort in the quit attempt [77,78].

The various game mechanics and elements were designed to appeal to users with different motivations. For example, Yee [79] identified 3 main components of player motivation: achievement (advancement and competition), social (socializing and relationships), and immersion (discovery, customization, and escapism). The experience-point system may appeal to achievers; the interaction with other players and a sense of connection with the dragon may appeal to socializers, and the rich opportunities for dragon customization and distraction games may appeal to players seeking immersion.

A more thorough description of the design and development of Inner Dragon is described elsewhere [41].

Measures

Outcome Measures

The study focused on outcomes within the domains of user engagement, abstinence from smoking, and user satisfaction and motivation.

User engagement measures included the two primary outcome measures of (1) the total number of unique app sessions from enrollment through 8 weeks after the user’s initial quit date, measured as the number of times the app was opened; and (2) the mean duration of app sessions, in minutes, through 8 weeks after the quit date; The secondary outcome measures of (3) the number of unique days with ≥1 app session, and (4) the proportion entering Inner Dragon, and the tertiary outcome measures of (5) program adherence, measured as the total number of times selected “core” features of the original Smoke Free app were used (reported a craving, recorded a diary entry, completed a mission, read a tip, and used the chatbot); and (6) the number of times using selected game features (opening gifts, breathing exercise, cleaning the dragon, feeding the dragon, memory minigame, using the customization menu to change appearance, and reading the dragon instruction guide). All engagement outcomes were passively collected by the app. A new session was defined as opening the app after at least 30 minutes of inactivity.

Smoking abstinence measures included the secondary outcome measures of (1) the proportion of participants who reported abstaining during the past 7 days at the 2-month follow-up assessment (self-reported 7-day point-prevalence abstinence), (2) the proportion of participants who reported abstaining during the past 30 days at the 2-month follow-up assessment (self-reported 30-day point-prevalence abstinence), and the tertiary outcome measures of (3) biochemically verified 7-day point-prevalence abstinence at the 2-month follow-up assessment, obtained from uploaded results from a self-administered salivary cotinine test (Alere iScreen Oral Fluid Device; Abbott Laboratories); and (4) repeated 1-day point-prevalence smoking abstinence, measured as the mean proportion of days each participant self-reported having abstained in the last 24 hours, collected via a pop-up box that appeared the first time each day a participant opened the app throughout the 8-week, postquit date period. Self-reported 7-day point-prevalence abstinence followed the Russell Standard, allowing for <5 slips [80]. Those who reported using nicotine replacement therapy (NRT) or electronic nicotine delivery systems (ENDSs) within the prior 7 days were coded in the main analysis as abstinent for the verified measures if no cigarettes were used [80,81].

User satisfaction and motivation measures included the secondary outcome measures of (1) satisfaction with the Smoke Free app (“I liked using the Smoke Free app”), reported on a 5-point Likert scale ranging from “not at all” (score=1) to “extremely” (score=5); (2) satisfaction with the Inner Dragon game (“I liked the dragon game”), reported on the same 5-point Likert scale, and the tertiary outcome measures of (3) rating of whether the person would recommend the assigned app to a friend on a 5-point Likert scale; (4) motivation to (stay) quit at the 8-week follow-up, reported on a 10-point scale from 0 (not at all motivated) to 10 (very motivated); and (5) digital therapeutic alliance, measured from the bonding and confidence subscales of the Mobile Agnew Relationship Measure, reported as a 16-point ordinal measure [42,70].

Covariates

The screening questionnaire collected demographic information on age, gender (woman, man, non-binary or other), race and ethnicity (recoded to non-Hispanic White, non-Hispanic Black, Hispanic, or other), household income category (US $0-US $19,999, US $20,000-US $39,999, US $40,000-US $59,999, US $60,000-US $79,999, US $80,000-US $99,999, and US $100,000 and more), and educational attainment (high-school diploma and less, some college or technical school, bachelor’s or associate degree, and graduate degree). Smoking characteristics at the screening included mean cigarettes per day, Fagerström Test for Nicotine Dependence (score of 0-10, where 10 is highly nicotine dependent) [82], number of past quit attempts, years since initiated, use of ENDS in last 30 days (yes or no), and use of NRT in last 30 days (yes or no). Baseline information also included frequency of video game use (not at all, less than once a month, at least monthly but not weekly, at least weekly but not every day, and every day).

Sample Size

Power calculations were based on an anticipated sample size of 500, with an assumed 70% retention (as observed in other app-based smoking cessation trials) [12,20]. We further anticipated a mean number of app sessions of 30 (SD 45.0) per participant in the control group, based on data from past Smoke Free users provided to the study team by Smoke Free. A sample size of 500 participants with 70% retention was estimated to detect a between-group difference in app sessions, one of the primary outcomes, of 13.5 at 80% power (α=.05).

Data Analysis

Main Analysis

The main analyses were conducted on an intention-to-treat basis using all randomized participants. For those outcomes measured continuously, including the 2 primary outcomes, we calculated the crude (unadjusted) difference in outcomes between each study group, using 1-tailed t tests of the difference in means to assess statistical significance at the 5% level. For those outcomes measured categorically, we calculated the mean difference in proportions corresponding to the risk difference and used a similar approach to assess statistical significance. For the abstinence outcomes, the intention-to-treat approach implied that those lost to follow-up were assumed to have resumed smoking (“missing=smoking”).

Outcome Trends

Next, we assessed visual changes over time in the number of users with a Smoke Free session, a key engagement outcome, and self-reported 1-day point-prevalence abstinence on an intention-to-treat basis (missing=smoking) and using complete cases. For each outcome, we plotted the mean for each study group by study day.

Intensity of Game Use

We assessed whether the intensity of game use was associated with more distal outcomes: total user engagement (total number of sessions), program adherence, and self-reported abstinence (at follow-up and repeated daily, assuming missing=smoking). This analysis, conducted among participants in the treated group only, entailed plotting estimates from local linear regressions of the number of sessions with game use on each of these outcomes, following the nonparametric approach proposed by Calonico et al [83]. The estimates included mean squared error-optimal bandwidth, 30 evaluation points. and a heteroskedasticity-robust nearest neighbor variance estimator with 3 neighbors.

Sensitivity and Exploratory Analyses

We examined subgroup differences across study groups for the primary engagement outcomes, in which we evaluated heterogeneity by estimating a stratified model for each covariate.

We conducted covariate-adjusted regression analyses to assess the robustness of our findings. We estimated the differences in each outcome by study group using linear regressions for continuous outcomes and logistic regressions for binary outcomes. The models adjusted for a range of prespecified potential confounders (described in the Covariates section earlier), including characteristics of participants’ sociodemographic background, smoking history, and frequency of video game use. Coefficients from the logistic regressions were expressed as risk differences, with an interpretation as a change in percentage points.

We repeated our analysis of verified abstinence—coding users of NRT or ENDS at follow-up as having not abstained. We also repeated our abstinence analysis using a complete case analysis that included only those participants who were successfully followed up. Whereas missing=smoking can lead to downward bias if loss to follow-up occurs for reasons other than relapse to smoking, complete case analysis may lead to upward bias if participants refuse to engage with follow-up because they have resumed smoking. As a further sensitivity analysis of the abstinence outcome, we estimated the treatment effect for self-reported 7-day point-prevalence abstinence from a pattern-mixture model that varied the informative missingness odds ratio between the outcome and an indicator for missingness, adjusting for covariates [84,85].

Adverse Events

We tabulated the number of serious adverse events by type for each study group, as reported in an open-ended text field in the follow-up questionnaire.

Analyses were conducted in Stata (version 18.0; StataCorp).

Ethical Considerations

Ethics approval was obtained from the University of California, San Francisco institutional review board (19-29335). All participants were informed about the study details and procedures before providing their consent electronically. Participation was strictly voluntary, and participants were informed about their right to withdraw from the study at any point without any consequences. The collected data were coded and deidentified. Participants’ identities were kept confidential and stored securely in password-protected files. Coded identifiers were given to participants after obtaining informed consent. No participants’ personal identifiers were used in analyses or report writing. Participants in both groups received US $20 for completing the baseline questionnaire, US $40 for completing the follow-up questionnaire, and US $40 for completing the saliva test, if requested. All payments were provided, via email link, as a choice of a Visa (Visa Inc) prepaid card or one of hundreds of gift cards through the Tango digital payment platform (Tango Card Inc).

Results

Sample Characteristics

Figure 1 depicts the study flow diagram. We screened 1181 new Smoke Free users from February 9, 2022, through March 16, 2022, to reach our goal of enrolling 500 eligible, consenting individuals for the RCT. Of the randomized participants, 95.8% (479/500; treated: 238/251, 94.8%, and control: 241/249, 96.8%) opened Smoke Free after completing the screening questionnaire, thereby completing the onboarding process within Smoke Free. Individuals who did not finish onboarding did not return to the app after the screening survey and did not learn their allocation group. The denominator used for the primary analyses was 479.

‎

There was a 61% (292/479) response rate for the 8-week follow-up questionnaire (treated: 154/241, 63.9% and control: 138/238, 58%), and a 71% (137/193) response rate for the saliva test (among those completing the follow-up questionnaire and reporting having quit within the prior 7 days), including 74% (73/99) of invited control participants and 68% (64/94) of invited treated participants. Nonresponse for the follow-up questionnaire and saliva test was unrelated to the study group assignment, according to t tests of the difference in means. However, response rates for the questionnaire and saliva test increased with the number of app sessions, and the response rate for the saliva test was higher for invited non-Hispanic Black participants (23/28, 82%) and lower for invited non-Hispanic White participants (93/136, 68.4%).

Demographic and smoking characteristics were balanced across study groups (Table 1; Table S1 in Multimedia Appendix 3 [7,8,10,12-20,86-95]). Participants were drawn from 48 states across the United States (Midwest: 126/479, 26.3%, Northeast: 90/479, 18.8%, South: 160/479, 33.4%, and West: 103/479, 21.5%). The median age of participants was 36 (IQR 29-47) years, with 19.8% (95/479) of participants aged ≥50 years due to the recruitment quota. Three-quarters (358/479, 74.7%) of participants identified as women. Most (368/479, 76.8%) were non-Hispanic White, with 10% (48/479) non-Hispanic Black or African American and 5.8% (28/479) Hispanic. Nearly 62.6% (300/479) had a household income below US $60,000, and 41.3% (198/479) had a household income <150% of the federal poverty level at the time. Most (277/479, 57.8%) did not hold a college degree. For smoking and gaming characteristics, participants were mostly moderate-to-heavy smokers (median of 15.0, IQR 10-20, cigarettes per day), had previously attempted to quit (median of 4, IQR 2-9, past attempts), and varied in terms of video game use during the prior 30 days (not at all: 116/479, 24.2% and every day: 162/479, 33.8%).

Main Intervention Effects

User Engagement

Compared with control participants, treated participants had 5.3 more sessions of Smoke Free app use during the 8 study weeks, although this difference in the primary outcome was not statistically significant (mean 29.6, SD 36.5 sessions vs mean 24.3, SD 37.9 sessions; P=.06; Table 2). Treated participants used the app for more minutes per session than control participants (mean 6.9, SD 5.4 min vs 6.2, SD 5.2 min; P=.047 for the primary outcome) and had more days with an app session (mean 14.3, SD 15.3 days vs 13.3, SD 11.9 days; P=.03 for the secondary outcome). Overall, treated participants used the Inner Dragon game for an average of 7.1 (SD 14.9) days, with at least some use of each of the game features (Table S2 in Multimedia Appendix 3).

Program adherence, as measured by the number of times using certain core features in the original Smoke Free app, was greater for treated participants than control participants (mean 29.4, SD 41.3 times vs mean 22.6, SD 35.6 times; P=.03 for tertiary outcome). The largest increase in core features for the treated group occurred for the number of diary entries completed (mean 12.1, SD 15.9 vs mean 9.2, SD 13.2; P=.01 for tertiary outcome), although use of all core features increased (Table 2).

Smoking Abstinence

Self-reported 7-day point-prevalence abstinence on an intention-to-treat basis (assuming missing=smoking) was 39% (94/241) for the treated group and 42.4% (101/238) for the control group (difference –3.4 percentage points; P=.45 for secondary outcome). The difference widened when considering complete cases only, such that 61% (94/154) abstained in the treated group and 73.2% (101/138) in the control group (difference –12.1 percentage points; P=.03).

Verified 7-day point-prevalence abstinence at 2 months similarly showed no advantage for the treated group, such that the intention-to-treat estimates were 19.5% (47/241) in the treated group and 22.7% (54/238) in the control group (difference –3.2 percentage points; P=.39). This included 8.3% (20/241) of participants in the treated group and 8% (19/238) of participants in the control group who were coded abstinent because they reported having used NRT or ENDS within the prior 7 days at the 2-month follow-up. Self-reported 30-day point-prevalence abstinence at 2 months was similar among the treated group (54/241, 22.4%) and the control group (50/238, 21%) on an intention-to-treat basis (difference 1.4 percentage points; P=.71 for secondary outcome).

Self-reported mean repeated 1-day abstinence was higher for the treated group (mean 17.3%, SD 25.6) than for the control group (mean 12.4%, SD 21.3) on an intention-to-treat basis (difference 4.9 percentage points; P=.01) and among complete cases (difference 10 percentage points; P=.01, tertiary outcome).

Satisfaction and Motivation

Satisfaction with the Smoke Free app on a 5-point Likert scale (1=not at all to 5=extremely) did not differ by study group among those who completed the follow-up questionnaire (mean 3.7, SD 1.1 treated group vs mean 3.7, SD 1.0 control group; P=.91 for secondary outcome). Additional questions regarding satisfaction with the app were also similar between study groups (Figure S2 in Multimedia Appendix 3). Satisfaction with the Inner Dragon game (mean 3.3, SD 1.4) was a bit lower in the treated group than for the app overall.

Motivation to (stay) quit at follow-up, measured on a 10-point scale (0=not at all motivated to 10=very motivated), did not differ significantly (mean 9.3, SD 1.3 treated group vs mean 9.3, SD 1.3 control group; P=.87 for tertiary outcome); however, a Qualtrics programming error led to missing data for 51% (148/290) of respondents. The digital therapeutic alliance also did not differ by the study group at follow-up (mean 12.7, SD 2.2 treated group vs mean 12.9, SD 2.4 control group; P=.75).

Outcome Trends

Days with any app sessions decreased for both study groups over time, from roughly 80% on the person’s quit day to roughly 20% on day 56 (Figure 2). The treated group retained slightly increased use throughout this period.

‎

While the share of participants who reported having abstained the prior day decreased for both study groups, the treated group retained its advantage over the control group throughout the study period when analyzing the data on an intention-to-treat basis (Figure 3A). When analyzing complete cases only, we observe that the percentage of participants who reported having abstained remained high throughout the study period, and the magnitude of the difference between study groups was not as great as when assuming missing=smoking (Figure 3B).

‎

Intensity of Game Use

Treated participants who had more sessions of game use had more app sessions overall, which is a mechanical relationship. More sessions of game use was also associated with increased program adherence (Figure 4B), increased mean repeated 1-day abstinence (Figure 4C), and 7-day abstinence at follow-up (Figure 4D). A 1 SD in sessions of game use (from 0-15 game sessions) was associated with a 4-fold increase in program adherence and an increase in self-reported 7-day point-prevalence abstinence at follow-up of 12.9 percentage points (from 33.8%, 95% CI 24.5%-43.2%, to 46.7%, 95% CI 25.2%-57.5%).

‎

Sensitivity Analyses

Subgroup Analyses

The effects of Inner Dragon on user engagement did not significantly vary with participant characteristics, according to subgroup analyses, although certain patterns emerged (Figure S6 in Multimedia Appendix 3). In unadjusted analyses, the treatment effect on the total number of app sessions was greater for younger participants (aged <50 years) and non-Hispanic White participants compared with non-Hispanic Black or Hispanic participants and decreased with household income. Furthermore, the effects were greater for those with low nicotine dependence, fewer (<4) past quit attempts, and use of e-cigarettes in the last 30 days. Intervention effects did not vary by video gaming history.

Covariate-Adjusted Analyses

Adjusting for participant characteristics (Table 1) as covariates, the differences in user engagement outcomes by study group were similar to, but slightly smaller in magnitude than, the crude differences for number of app sessions (coefficient 4.6, 95% CI –2.2 to 11.3; P=.18), average minutes per session (coefficient 0.8, 95% CI –0.2 to 1.9; P=.11), days with a session (coefficient 2.2, 95% CI –0.4 to 4.7; P=.10), and program adherence (coefficient 6.6, 95% CI –0.4 to 13.6; P=.06; Table S3 in Multimedia Appendix 3). The effect sizes for self-reported and verified 7-day and self-reported 30-day point-prevalence abstinence remained similar to the crude differences in magnitude. Estimates, on an intention-to-treat basis, were –2.4 percentage points (95% CI –11.1 to 6.4; P=.60) for self-reported 7-day abstinence, –2.7 percentage points (95% CI –10.0 to 4.6; P=.47) for verified 7-day abstinence, and 1.4 percentage points (95% CI –6.0 to 8.9; P=.70) for self-reported 30-day abstinence. Mean repeated 1-day abstinence remained higher for the treated group than the control group (difference 4.5 percentage points, 95% CI 0.2 to 8.7; P=.04) on an intention-to-treat basis (Table S3 in Multimedia Appendix 3).

Sensitivity of Abstinence Estimates

Coding NRT and ENDS users as not abstinent, verified 7-day point-prevalence estimates were similar to the main results (difference 4.3 percentage points, 95% CI –1.8 to 10.5; P=.16). Results from the pattern-mixture model indicated that the treatment effect estimates for abstinence were highly sensitive to assumptions about the nature of the missing outcomes (Figure S1 in Multimedia Appendix 3). Assuming missingness is random (equivalent to missing=smoking), there would be no difference in abstinence between groups, whereas assuming informative missingness, there would be reduced abstinence in the treated group compared with the control group.

Adverse Events

There were no serious adverse events reported during the trial. In total, 1.2% (3/241) reported adverse events in the treated group and 2.9% (7/239) in the control group. Adverse events were feelings of withdrawal, craving, or irritability (1 report in the treated group and 4 in the control group); being anxious or depressed (1 report in the treated group and 3 in the control group); and constipation (1 report in the treated group and 0 in the control group).

Discussion

Principal Findings

Our RCT assessed the efficacy of the novel Inner Dragon game module integrated into the Smoke Free app for promoting user engagement. The study of 479 participants revealed that treated individuals exhibited some increased user engagement metrics, including a (nonsignificant) 21.8% increase in the number of app sessions and a significant 20.2% increase in days of app use. Engaging with Inner Dragon led users in the treated group to use core features of the original Smoke Free app significantly (30.1%) more often, and this increase in smoking cessation program adherence implies that game use complemented (“crowded in”) rather than replaced (“crowded out”) use of the nongame content. User engagement is a fundamental concern for mobile health apps [22-24], and our findings support the role of gamification as a key driver of user engagement.

Smoking abstinence outcomes showed a nuanced picture, with self-reported 7-day point-prevalence abstinence favoring the control group and repeated 1-day abstinence favoring the treated group. In exploratory analyses, we found that this difference was driven in part by treated participants with missing point-prevalence abstinence data having much higher mean repeated abstinence than control participants with missing point-prevalence abstinence data (mean 54.5% SD 5.5 vs mean 35.4% SD 4.9, difference 19.1 percentage points; P=.01). Nevertheless, improving user engagement and program adherence metrics did not definitively produce higher smoking abstinence rates in this trial. One possibility is that gamifying the quit process may have undermined participants’ focus on the higher purpose of quitting smoking. Further research into features of the game, user characteristics, and the original app may be warranted to understand the relationship between user engagement and smoking abstinence.

Finally, the study found a positive association between the intensity of game use and both program adherence and smoking abstinence, suggesting that game-driven engagement may support downstream outcomes.

Comparison With Previous Studies

In the context of the existing literature on smartphone apps for smoking cessation, our findings align with studies emphasizing the importance of user engagement in digital interventions [22-24]. Notably, the increased engagement observed in our study was associated with positive outcomes such as greater program adherence in the form of higher core feature use. To our knowledge, the link between user engagement and program adherence has not been explored with regard to games for smoking cessation, in part because previously evaluated games have been stand-alone products [96].

We found some evidence of effect modification, such that user engagement was greater for younger and lower-income users. Our results echo the call for personalized interventions, consistent with research suggesting the impact of individual characteristics on engagement and outcomes in digital health interventions [97]. Furthermore, our exploration of gamification elements in the Inner Dragon game aligns with literature highlighting the potential of gaming features to enhance engagement and promote behavior change in smoking cessation apps [96]. Prior studies of smartphone-based games for smoking cessation have found mixed evidence regarding smoking abstinence [19,20,38,98], although study quality in the broader literature has generally been deemed fair or poor with important methodological limitations [98]. We found that the bundled game intervention as a whole contributed to the effects of Inner Dragon on increased user engagement. We were not able to isolate the effect of any individual feature of Inner Dragon; exploratory analyses revealed that participants’ use of most key game features was moderately or strongly positively associated with the total number of app sessions (range: 0.39-0.74), and each game feature was strongly or very strongly positively associated with other game features (range: 0.53-1.00; Figure S5 in Multimedia Appendix 3). A future study might use a factorial design to identify the most important game features for stimulating user engagement and abstinence.

Our study supports the integration of personalized strategies, such as games in digital interventions, tailored to individual characteristics and broadly appealing to users with different motivations (eg, achievement, socializing, and immersion), to optimize user engagement and intervention effectiveness. Emerging capabilities of artificial intelligence may facilitate this process [99]. Relatedly, self-paced initiatives such as gamified smoking cessation apps constitute an alternative to traditional smoking cessation interventions that often depend on advanced scheduling. Many researchers have sought ways to leverage technological advancements that enable the delivery of just-in-time, tailored support for behavior change, although large-scale studies on the effectiveness of smoking cessation are lacking [100].

Comparing our abstinence rates to previous studies of smoking cessation apps is difficult due to important differences in study design, such as intervention delivery, comparator, follow-up period, abstinence measure, and the use of bioverification. Focusing on 14 RCTs with follow-up of 1 to 3 months, abstinence ranged widely from 1.6% (234/14,228) to 81.8% (9/11) in the treated groups and 0.9% (124/13,884) to 66.2% (190/287) in the control groups, and intention-to-treat intervention effects ranged from –0.9 to 11.5 percentage points (Table S4 in Multimedia Appendix 3). We observed abstinence within these ranges, although abstinence in our control group was relatively high (101/238, 42.4%), suggesting that the original Smoke Free app was a particularly effective comparator group on its own. Our study is also distinguished from other app-based studies for having collected high-frequency (daily) abstinence data as surveillance within a smoking cessation intervention; however, there is, related literature on just-in-time adaptive interventions that collect high-frequency data with the aim of identifying smoking triggers and delivering real-time feedback [100].

Strengths and Limitations

This study has important strengths. First, this study is one of the largest randomized trials of a smartphone-based intervention involving games or gamification for smoking cessation. Second, to our knowledge, it is the first study to assess a game that is integrated into an existing smoking cessation app. By integrating Inner Dragon into one of the most downloaded smoking cessation apps, which also has established effectiveness, we were able to evaluate the added benefit of the game module for user engagement. The game intervention can be readily scaled to the 800,000 users who download Smoke Free each year; Inner Dragon is planned to become available to general users of Smoke Free starting in 2024. Third, passive collection of use data ensured that our primary outcome data were complete, accurate, and able to be tracked longitudinally. Fourth, the availability of high-frequency, longitudinal abstinence data was unique for studies of smoking cessation games and uncommon for studies of smoking cessation interventions more generally [101].

However, this study has several limitations. First, the short-term follow-up limits our ability to assess sustained abstinence outcomes. Second, the generalizability of our findings may be constrained by the demographic composition of the app-recruited study population. It is noteworthy that our study drew from a general population of users of the Smoke Free app, a diverse user base in a real-world context. Yet, the use of the Smoke Free app may not be representative of all smoking cessation apps, and the rapidly evolving landscape of digital interventions may influence the generalizability of our findings over time. Third, our game intervention includes a bundle of interlocking features, and we cannot disentangle their individual effects. Fourth, while our study was adequately powered for our primary outcomes, we may have been underpowered for some secondary outcomes, such as smoking abstinence, as well as for subgroup analyses.

Conclusions

In conclusion, our study contributes valuable insights to the growing body of literature on digital smoking cessation interventions involving serious games, emphasizing the role of user engagement. The positive associations between game use with program adherence and smoking cessation outcomes underscore the potential of gamification interventions in promoting behavior change. Future research should extend follow-up durations, refine personalized interventions, evaluate effectiveness in a real-world setting, and address methodological challenges, such as missing data, to guide the development of evidence-based digital health interventions for smoking cessation.