Mobile Applications for Stroke Prevention: A Survey of Physicians’ Perspectives

Mobile Applications for Stroke Prevention: A Survey of Physicians’ Perspectives

Douglas Halket, BS¹, Jonathan Singer, MA², Clotilde Balucani, MD, PhD³, Dimitre Stefanov, PhD³, Steven R. Levine, MD, FAHA, FAAN, FANA⁴

¹SUNY Downstate College of Medicine, MSC, Brooklyn, NY; ²University of Nevada, Reno, Clinical Psychology, MSS, Reno, NV, ³SUNY Downstate Medical Center & Stroke Center, MSC, Brooklyn, NY; ⁴University Hospital of Brooklyn, Associate Dean for Clinical Research & Faculty Development, Downstate Medical Center & Stroke Center

Corresponding Author: douglas.halket@downstate.edu

Journal MTM 6:3:7–13, 2017

doi:10.7309/jmtm.6.3.3

---

---

Introduction

Stroke is the leading cause of preventable disability in the United States¹. Each year, approximately 795,000 Americans suffer a stroke¹. Many of these strokes are preventable. According to a case-control study of 6,000 individuals, 90% of strokes can be attributed to just 10 modifiable risk factors, and targeted interventions to reduce blood pressure and cigarette smoking as well as to promote physical activity and healthy diet could substantially reduce the risk of stroke². There are already many mobile applications that individually target each one of these interventions, although not necessarily in the specific context of stroke prevention³.

Healthcare related mobile technology has expanded rapidly over the last several years. In 2015, there were 165,000 medical and health related apps for sale in the Google and Apple store⁴, up from 40,000 in 2012⁵. However, little is known about the prevalence and nature of mobile application adoption in patient centered clinical practice, let alone about its effectiveness in changing healthcare outcomes. While there are many mobile applications in fields such as diabetes management, there are still usability and integration issues among almost all of them⁶. A 2014 study found 93 mobile application for iPhone and Android regarding general stroke information, although many of these were lacking scientifically valid information⁷.

There have been relatively few randomized controlled trials involving mobile applications, although some have shown promise. A systematic review of 9 studies assessing mobile application effectiveness in managing cardiovascular disease, lung disease, or diabetes mellitus found 3 studies where a mobile application intervention showed a statistically significant clinical improvement⁸. Additionally, there has been at least one quality improvement project studying mobile technology assistance in blood pressure control for stroke survivors⁹, although there have been no large scale randomized control trials specifically involving stroke prevention. Mobile applications have the potential to change the healthcare landscape, leading to improved outcomes while reducing cost. If mobile technology is going to be implemented in a meaningful way, it is imperative that it be grounded in clinical evidence.

The current study is the first step in examining the role that mobile technology plays in primary care physicians’ (PCPs’) practices. The purpose of this study is to examine the prevalence of PCPs using mobile applications in patient care. More specifically, to study if and how PCPs use mobile applications for stroke prevention. Also, we evaluated preferences and attitudes of PCPs toward the use of mobile applications in their practices with specific focus in stroke management, and how these preferences vary over physician and practice demographics. We hypothesize that younger physicians practicing in wealthier areas are more likely to use and recommend mobile applications.

Methods

Current PCPs in the New York City area specializing in Internal Medicine, Ob/Gyn, and Family Medicine were surveyed in person.

Participants: A convenience sample of 86 physicians (46 females) ranging from 25 to 68 years of age participated in this study, which was approved by the SUNY Downstate IRB. Participants were surveyed in person and were identified at SUNY Downstate grand rounds and the annual meeting of the New York Academy of Family Physicians. The mean age was 37 ± 12 SD. Forty one percent of the participants were attendings and the remaining 59% were residents. Sixty (70%) were Family Medicine specialists, 14 (16%) were in Ob/Gyn, and 12 (14%) were in Internal Medicine. These three specialties were selected because they provide the primary care lead in general preventative care, and the study focus is on actually managing stroke risk factors. Neurologists, despite managing patients after stroke, do not generally provide the ongoing lead in preventative care unless a patient has a separate neurological issue, and so were excluded from the study.

Measures: The standardized survey consisted of demographic questions and practice characteristics including gender, specialty, resident/attending, type of practice (private independent, private partnership, community or university hospital), years of experience (less than 3, 3 to 10, 11 to 20, greater than 20), patients seen per month (less than 30, 30 to 100, greater than 100), median income of practice zip code as determined by census data (less than $40,000 or greater than $40,000), and 11 questions on mobile application use (table 1).

Table 1: Demographic Summary of Study Subjects

Data analysis: Fisher’s exact test was used to test for association between demographic variables and whether or not the physicians used mobile applications in any patient care setting. Cochran–Armitage trend test was used to determine if higher levels of the ordinal predictors (age groups, experience and volume of patients) were associated with higher prevalence of mobile technology use. P values less than 0.05 were considered significant. All participants answered the primary question on mobile application use in patient care. Missing data in other, secondary questions were not factored into the analysis as we were not adequately powered for other specific analyses. A sample size of 87 physicians was determined to have 80% power to detect an odds ratio of 0.4 for the relationship between age and the use of mobile applications using a significance level of 0.05. The odds ratio of 0.4 corresponds to a hypothesized effect of physician age on mobile application use. This odds ratio is based on an expected 30% use of mobile applications by physicians at age 50, compared to 15% for physicians at age 63. The data were analyzed using SAS version 9.4 (SAS Institute Inc., Cary, N.C.)

Results

Of the 86 physicians surveyed, 74% reported using mobile applications in patient care, whether for their own use or in recommending to patients. Physicians reported using mobile applications to manage stroke risk factors 25% of the time, while 77% expressed interest in new apps to help their patients manage these risks. Forty one percent of physicians surveyed strongly agreed that mobile applications are useful in providing patient care, while 49% simply agreed and 0% disagreed.

Table 2 shows the most important determining factor, with 82% of physicians with less than 3 years practice experience using mobile apps, 78% of physicians with 3 to 10 years, 60% of physicians with 11 to 20 years, and 58% of physicians with greater than 20 years experience (p=0.045). Other demographic factors, including median income of practice zip code were not significant.

Table 2: Mobile Application Usage Responses

Age did not have an effect on whether or not physicians agree that mobile applications are useful in providing patient care (p=0.27; table 3). Position, type of practice, and volume of patients were not associated with higher use of mobile technology use (p=0.80, 0.49, 0.33, respectively).

Table 3: Usage of Mobile Applications in Patient Care by Demographics

Discussion

Previous studies have shown that there are a large number of mobile applications available to patients and physicians, particularly to manage stroke risk factors^3,4,6–8. This study builds on this knowledge in an attempt to determine how physicians view mobile application use in the primary care clinical setting, particularly in stroke prevention. Our data revealed that of a sampling of mostly urban PCPs, 74% are already using mobile applications in clinical practice and an even greater portion, 90%, believe that mobile applications belong in patient care. Although PCPs with more experience were less likely to use mobile applications, the findings suggest a wide acceptance across physician age groups and practice characteristics. Income of practice zip code was not a significant predictor in whether or not PCPs recommend mobile applications to patients, and neither was PCP age. Physician experience was potentially important, with less experienced PCPs more likely to recommend mobile applications than their more experienced colleagues. These findings suggest the adoption of mobile application use in clinical practice will likely continue to increase in the coming years.

Some physicians, 34% of the respondents, expressed concerns over mobile technology use in healthcare. These concerns included ease of use, confidentiality, determining which app is best, time limitations, and integration with Electric Health Records (EHR). Some of these concerns, such as determining which apps are best, can be solved with further study. Confidentiality and EHR integration issues are more complex but are also potentially manageable in the future⁵.

There was a major disparity between PCP-expressed interest in mobile applications (76%) compared with their actual use of mobile applications to manage stroke risk factors (25%). This gap between interest and use can be bridged by either the development of new mobile applications specifically tailored to manage stroke risk factors or the development of a tool to assist patients and physicians in selecting from already existing mobile applications. The development of such a tool would likely require a systematic method for determining which mobile applications are effective in the patient care setting. Mobile applications that can assist in managing chronic risk factors for stroke can then be identified, potentially leading to an inexpensive and effective method for reducing one of the largest causes of morbidity and mortality in the United States^1,2.

There were several limitations to the study. First, the sample size was one under the projected number, as the data were collected from physicians in discrete groups (grand rounds, etc.) rather than from individuals. Second the data were slightly skewed by specialty and position. We recruited a large percentage of Family Medicine attendings because of the ease of recruiting them at their annual meeting. Many of the Ob/Gyn and Internal Medicine respondents were residents because they were more likely to attend grand rounds and stay to fill out a survey. There were few respondents from high income zip codes. We acknowledge the potential lack of generalizability to other specialties, including neurologists who may participate in stroke prevention but generally do not lead and primarily manage this effort in their patients.

Conclusions

The majority of PCPs have either already adopted mobile applications in their clinical practice or believe that mobile applications belong in patient care. However, the implementation of mobile applications specifically for stroke prevention purposes is still limited. Concerns about mobile application use were limited to a small number of physicians and a significant number expressed interest in a new application to manage stroke risk factors. There is PCP interest in a mobile application specifically tailored to manage stroke risk factors that patients and physicians can use together and encompasses several aspects of stroke prevention. Since many physicians are already using mobile applications or support their use in patient care, it is likely that mobile applications will have a place in clinical practice in the future, and specifically in stroke prevention.

Acknowledgements

Supported by a SUNY Alumni fund medical student research scholarship to DH.

Disclosures

All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: DH received support from a SUNY Alumni grant, no other support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.

References

1. Mozaffarian D, Benjamin EJ, Go AS, Arnett DK, Blaha MJ, Cushman M, et al. Heart Disease and Stroke Statistics-2016 Update: A Report From the American Heart Association. Circulation. 2016;133(4):e38–360.

2. O’Donnell MJ, Xavier D, Liu L, Zhang H, Chin SL, Rao-Melacini P, et al. Risk factors for ischaemic and intracerebral haemorrhagic stroke in 22 countries (the INTERSTROKE study): a case-control study. Lancet (London, England). 2010;376(9735):112–23.

3. Mosa ASM, Yoo I, Sheets L. A Systematic Review of Healthcare Applications for Smartphones. BMC Medical Informatics and Decision Making. 2012;12:67.

4. Misra S. New report finds that more than 165,000 mobile health apps now available, takes close look at characteristics & use: iMedicalApps; 2015 [updated September 17, 2015. Available from: http://www.imedicalapps.com/2015/09/ims-health-apps-report/.

5. West D. How mobile devices are transforming healthcare. Issues in technology innovation. 2012;18(1):1–11.

6. El-Gayar O, Timsina P, Nawar N, Eid W. Mobile Applications for Diabetes Self-Management: Status and Potential. Journal of Diabetes Science and Technology. 2013;7(1):247–62.

7. Dubey D, Amritphale A, Sawhney A, Amritphale N, Dubey P, Pandey A. Smart phone applications as a source of information on stroke. Journal of stroke. 2014;16(2):86–90.

8. Whitehead L, Seaton P. The Effectiveness of Self-Management Mobile Phone and Tablet Apps in Long-term Condition Management: A Systematic Review. Journal of medical Internet research. 2016;18(5):e97.

9. Ovbiagele B, Jenkins C, Patel S, Brunner-Jackson B, Anderson A, Saulson R, et al. Mobile health medication adherence and blood pressure control in recent stroke patients. Journal of the neurological sciences. 2015;358(1–2):535–7.

Read More

Scope of Mobile Phones in Mental Health Care in Low Resource Settings

Scope of Mobile Phones in Mental Health Care in Low Resource Settings

M Sood¹, RK Chadda² K Sinha Deb³, R Bhad⁴, A Mahapatra⁴, R Verma³, AK Mishra³

¹Additional Professor; ²Professor; ³Assistant Professor; ⁴Senior Resident
Department of Psychiatry, All India Institute of Medical Sciences, Ansari Nagar, New Delhi 110029, India.

Corresponding Author: soodmamta@gmail.com

Journal MTM 6: 3:43–47, 2017

doi:10.7309/jmtm.6.3.7

---

---

Introduction

In the last few years, there has been a remarkable spread of mobile technology in low and middle-income (LAMI) countries, with its penetration being much higher than the general infrastructure. For example, in India, there are 943.9 million wireless telephones with a teledensity of ~75%, and the mobile phones constitute about 97% of the total telephones¹. Mobile technologies, with an easy and wider availability, portability, being self powered, increasingly better computational capacities and decreasing costs, user familiarity and internet connectivity², have opened up new alternatives for health care delivery that can be delivered through the existing infrastructure as people carry mobiles with them all the time.²

Recent studies from high-income (HI) countries have reported that a number of people use mobile phones to search for health related information. Most patients with mental illness own mobile phones, use it for activities other than spoken conversations like sending emails, web browsing and social networking, and mobile phones have been explored as a potential aid in mental health services³. Studies from HI countries have demonstrated use of mobile phones in delivering psychosocial interventions (crucial for recovery) like provision of health information, prompts for medications, reminders, self-monitoring, and practice of skills in real world situations⁴.

In India, mental health resources for psychosocial interventions are meagre and are rarely applied in the care of patients with mental disorders⁵ as there is a gross deficiency of mental health resources (0.2 psychiatrists, 0.03 clinical psychologists, 0.05 psychiatric nurses, and 0.03 social workers per 100,000 of the population^6,7. In this context, mobile technologies have a potential to become an important mental health care service link between the meagre mental health services and the unfulfilled mental health care needs of the vast majority of unreached patients and caregivers, and can be used for a range of indications in mental health like increasing awareness, training, linkages between services, clinical services and research.

We are planning to develop a mobile based intervention framework for imparting psychosocial interventions to the patients attending psychiatric services. Before making mobile based interventions suitable for the needs of the patients, it is important to know whether the patients with mental illness are using mobile phones as there is absence of research in this field from India. The present study was conducted to find out the frequency of mobile phone use among patients with mental illnesses, and to assess feasibility of using mobile phones in improving service delivery and delivering educational messages in them.

Methodology

The study was conducted in psychiatry outpatient services at the All India Institute of Medical Sciences, New Delhi, India. The service runs a walk-in clinic, where all the first contact patients are seen, and a follow up clinic where the old patients already on treatment from the service are seen. Every fifth patient aged 18-60 years, visiting the walk-in clinic, and every fifth patient visiting the follow up clinic over a period of three months (from February – April, 2015) were recruited for the study.

The subjects were explained the purpose of the study. A written informed consent was taken. In patients with psychotic disorders, who were unable to give consent, consent was taken from the accompanying relative. The data was collected using a semi-structured questionnaire (attached as Appendix), prepared for the study by the investigators. The study was approved by the Institute ethics committee.

The data was analysed using descriptive statistics, reported as means and standard deviations for continuous variables and percentages for discrete variables.

Results

A total of 350 subjects were recruited for the study, 205 (58.6%) males and 145 (41. 4 %) females. Two hundred and thirty five (77.1%) subjects were from the follow up clinic and 115(32.9%) were from the walk in clinic. Mean age of the subjects was 33.3 (±11.5) years. About 40% of the subjects had received 10 years of formal education, one fifth had studied upto 12^th standard, and about 30% were graduate. Mean duration of illness was 6.2 (± 7.6) years, and mean duration of treatment was 1.5 (± 3.5) years. Common diagnosis included neurotic, stress related and somatoform disorders (47.0%), mood disorders (32.0%), schizophrenia and related disorders (14.3%), and disorders due to psychoactive substance use (4.0%).

Three hundred and seven (87.7%) subjects reported using mobile phones regularly. Most of the subjects used mobile phone for making and receiving phone calls, and about two third also used it for sending and receiving short text messages. About half of them used clock and alarm functionalities, two fifth also used it for recreational activities, and around 30% used for accessing social network sites (Table 1)

Table 1: Mobile phone use and its utility as aid to psychiatric treatment Application of mobile phone in day-to-day life (n=307)

Most of the subjects reported that mobile phones could be used as an aid to treatment for psychiatric disorders. This included use as a reminder for appointments (90.4%), and to take medications (72.3%). About half of the subjects suggested that mobile phones could be used for recording and reporting of side effects. Forty two percent of the subjects reported that mobile phones could be used for receiving educational information related to their mental illness. Only 17% of the patients felt that these could be used for imparting psychological treatments (Table 1).

More than 70% of the mobile phone using subjects expressed that the phones could be used to receive educational messages regarding any precautions to be taken (70.5%), activities and exercises (55.4%), information about their mental illness (36.5%), dietary advice (26.9%) and information about stress reduction techniques (Table 1).

In a single open-ended question about any other use of mobile phone related to psychiatric treatment, only 31 (11.4%) subjects provided a suggestion. Sixteen subjects felt that mobile phones can be used to give feedback regarding treatment. Fifteen subjects felt that these can be used for sharing experience regarding treatment through common patient group on social networking sites.

Discussion

More than 85% of the patients with mental health problems in our study were using a mobile phone. The mobile phones was being used for a range of activities besides making phone calls. Most subjects opined that it could also be used as an aid in the treatment.

All of our subjects used mobile phone for making and receiving phone calls, but over two third also used it for sending and receiving short text messages. The phone was also used for other functions like clock and alarm by about half of the subjects, for recreation by about two fifths, and for accessing social network sites by 30% of the subjects. Earlier studies have also reported the most common uses of mobile phones in patients with severe mental illnesses as for making phone calls, texting, and internet³.

Regarding the potential usage of mobiles phones in treatment, most (90%) of the subjects reported that these can be used to remind them for appointments. Majority of the subjects indicated use of mobile phones in pharmacological treatment as reminder for taking medications and reporting side effects. However, only less than half of the subjects indicated that mobile phones can be used for receiving educational information related to their mental illness and even fewer subjects felt that these can be used for imparting psychological treatments. This finding is not unusual as majority of the patients with mental illnesses receive pharmacological treatment with minimal psychosocial interventions due to lack of resources for the latter⁵. However, when asked specifically about the kind of educational messages they wanted to receive, majority of them listed only psychosocial interventions. About three fourth of the subjects reported that they wanted to receive messages regarding any precautions to be taken regarding illness or medications, about half of them wanted messages regarding activities and exercises, and one third wanted information about their mental illness, and a quarter each wanted information regarding dietary advice and stress reduction techniques. This shows that the mobile phones appear to have a high potential of use in the mental health care settings.

Our results are in line with the studies from HI countries which have reported that n 72%-97% of patients with mental illnesses and substance use disorders own a mobile phone^{2, 8, 9}. The results are not surprising since compared to an average 2008–2012 growth rate of mobile subscriptions of just 10.15% in the HI countries, the growth rate was much higher (75.07%) in South Asia driven mainly by growth in India¹⁰.

The study had a limitation of being conducted in a tertiary care setting in a big city and hence the findings may not be generalizable to other settings. We did not screen specifically for the use of smart phones, since most of our subjects use ordinary phones.

The study concludes that most of the patients with psychiatric disorders attending outpatient services in India use mobile phones, and welcome its use as a treatment aid. The mobile phones offer a potential to be exploited in India with limited human resources in mental health and a potential use in psychosocial treatment.

References

1. ANNUAL REPORT (2014). Department of Telecommunications, Ministry of Communications and Information Technology, Government of India, New Delhi. Available at http://www.dot.gov.in/sites/default/files/Telecom%20Annual%20Report-2012-13%20%28English%29%20_For%20web%20%281%29.pdf (accessed on 12 Aug 2015)

2. Zhang MWB, Ho CSH, Cheok CCS, Ho RCM. Smartphone apps in mental healthcare: the state of the art and potential developments. BJ Psych Advances, 2015;21(5):354–8.

3. Ben-Zeev D, Davis KE, Kaiser S, Krzsos I, and Drake RE. Mobile technologies among people with serious mental illness: opportunities for future services. Adm and Policy Ment Health 2013;40(4):340–3.

4. Harrison V, Proudfoot J, Wee P P, Parker G, Pavlovic DH, Manicavasgar V Mobile mental health: Review of the emerging field and proof of concept study. J Ment Health 2011;20(6):509–24.

5. Sood M, Chadda RK. Psychosocial rehabilitation for severe mental illnesses in general hospital psychiatric settings in India. BJPsych International 2015;12:47–8.

6. Chadda RK, Prashanth R (2015) Allied Mental Health Professionals: clinical Psychologists, Psychiatric Nurses and Psychiatric Social Workers: Availability and competency. In Mental Health in South Asia: Ethics, Resources, Programs and Legislation (Eds JK Trivedi & A Tripathi) Springer (India) pp. 221–32.

7. The World Health Report: 2001: Mental in Low and Middle-Income (LAMI) Countries, World Health Organization, 2001.

8. Crankshaw T, Corless I B, Giddy J, Nicholas P K, Eichbaum Q, Butler L M. Exploring the patterns of use and the feasibility of using cellular phones for clinic appointment reminders and adherence messages in an antiretroviral treatment clinic, Durban, South Africa. AIDS Patient Care STDS 2010;24(11);729–734.

9. Milward J, Day E., Wadsworth E, Strang J, Lynskey M. Mobile phone ownership, usage and readiness to use by patients in drug treatment. Drug Alcohol Depend 2015;146(1):111–5.

10. Farrington C, Aristidou A, Ruggeri K. mHealth and global mental health: still waiting for the mH2wedding? Globalization and Health 2014;10:17.

Appendix

Feasibility of using mobile phones in improving service delivery and creating health awareness in patients with mental disorders

(Please tick mark the answers)

DateRegn No.Name

AgeGender Education

Address:

Mobile:E mail (if any):

Diagnosis

Duration of Illness:

Duration of treatment at AIIMS:

1. 1. Do you have a mobile phone?Yes/No

2. 2. For what all functions do you use mobile phone? (Kindly mark as many as applicable)

   • • Making & receiving calls

   • • Sending & receiving messages

   • • Clock & alarm

   • • Contact list

   • • Any other

3. 3. Do you think that it can be used as a helping aid in your treatment? Yes/No

4. 4. If yes, kindly tell, how? (Kindly mark as many as applicable)

5. • • Reminder for appointment

   • • Reminder to take medicines

   • • Recording & reporting of side effects

   • • Receiving education messages related to your illness

   • • Psychological treatment

   • • Any other

6. 5. What kind of educational messages you would like to receive?

   • • Any precautions to be taken

   • • Activities or exercises

   • • Dietary advice

   • • Any other

Any suggestions, you would like to make

Read More

Usability of a Mobile Application for Patients Rehabilitating in their Community

Usability of a Mobile Application for Patients Rehabilitating in their Community

Jeanette R. Little, BS, MS¹, Holly H. Pavliscsak, BS, MHSA², Mabel Cooper, RN, BSN, CCRC³, Lois Goldstein, RN, BSN³, James Tong, BS, MBA⁴, Stephanie J. Fonda, PhD⁵

¹Mobile Health Innovation Center, Telemedicine and Advanced Technology Research Center, United States Army Medical Research and Materiel Command; ²The Geneva Foundation, Assigned to the Mobile Health Care Innovation Center (MHIC), Telemedicine and Advanced Technology Research Center (TATRC), United States Army Medical Research and Materiel Command (USAMRMC); ³The Geneva Foundation, Assigned to the Mobile Health Care Innovation Center (MHIC), Telemedicine and Advanced Technology Research Center (TATRC); ⁴IMS Health, Government Solutions; ⁵The Geneva Foundation, Assigned to Telemedicine and Advanced Technology Research Center (TATRC)

Corresponding Author: jeanette.r.little.civ@mail.mil

Journal MTM 6:3:14–24, 2017

doi:10.7309/jmtm.6.3.4

---

---

Introduction

In 2008, the United States Army Medical Department, through the Telemedicine and Advanced Technology Research Center (TATRC), developed and tested ‘mCare’, a secure mobile application for providing care management to wounded men and women in the military who were rehabilitating in their communities. The mCare application facilitates and enhances contact between the person rehabilitating in the community and their case managers and other care team members. It sends to the user’s phone: appointment reminders; health and wellness tips; community resources and administrative announcements; and questionnaires on various aspects of the user’s well-being. It can also transmit the user’s responses to questionnaires back to their care team. The mCare application was developed to support Service Members with mild traumatic brain injury (mTBI) or post-traumatic stress (PTS) rehabilitating in the community, with the goal of eventually expanding the application to support other care management use cases.

To assess the impact of mCare on users’ quality of life and adherence to appointments and rehabilitation goals, the TATRC team conducted a randomized controlled trial (RCT) in which participants were recruited and allocated to receive standard care management or standard care management plus mCare. As part of that trial, the TATRC team assessed usability of the mCare application, as perceived among those randomized to receive it. This article reports on the perceived usability of the application overall and how it relates to important characteristics of the mCare cohort, namely behavioral health, mTBI and PTS status, technology (e.g., type of phone), environment (i.e., population size of the participants’ community), and usage of the application. Although the TATRC team did not apply an iterative user-centered design model per se,² and this is not a usability test performed in a laboratory environment for the purpose of finding problems,^3,4 post-study usability evaluations such as this one ­provide important information on the translation and sustainment of new technologies into use outside of the scope of a study, from the perspective of users who have become ‘experts’ on the technology over time.

Methods

mCare application

The mCare application has been previously described.¹ In brief, it is a secure, bi-directional, application connected to a backend system that facilitates contact between Service Members and their care team members. The mCare application is remotely activated on a user’s mobile device. The system uses text messages to notify users when there is new information to view and respond to within the application. This new information includes daily questionnaires, appointment reminders, health tips, and administrative information. The backend system also provides a secure website that tracks all data from the mobile devices and generates escalation triggers (as needed) to the care team. To access the new information, the user has to enter their mCare personal identification number, which is part of a secure two-factor authentication process that makes mCare unique from most mobile health applications.

For the RCT, mCare sent a daily questionnaire, three health and wellness tips per week, administrative announcements, and appointment reminders 24 hours before, 90 minutes before, and 24 hours after each clinical encounter. The questionnaires covered one functional topic area per day (General Status, Pain Status, Energy and Sleep Status, Anger Management, Relationship Status, Transition Goal Status, and Mood Status) and each functional topic was repeated weekly. The mCare application also sent a Weight Status questionnaire once per month. The mCare questionnaires can be modified for other populations, as appropriate, making it a generalizable adjunct to standard care management for a variety of conditions.

Study Design

Injured Service Members newly assigned to four participating Community-Based Warrior Transition Units (CBWTUs) (now called Community Care Units [CCUs]) were recruited to participate voluntarily in the RCT testing the efficacy and usability of mCare (n=182). Among other things, eligibility criteria included having a cell phone, proficiency in the use of that cell phone, and demonstrated ability to send and receive text messages. The data for this analysis are from the intervention arm (n=95). Henceforth, this article refers to participants in the intervention arm only.

The purpose of the CBTWUs was to help Service Members rehabilitate in their home communities and return to duty. If a participant’s assignment to one of the four participating CBWTUs ended (referred to as ‘out processed’), their time in the study concluded early, because being at a CBWTU was a criterion for study participation. In this study, Service Members left their CBWTU for the following reasons: ‘Release from Active Duty’ (n=28); ‘Separation’ (n=1); or returned to a Military Treatment Facility (MTF) (n=3). Release from Active Duty applied to soldiers who were National Guard and Reservists and indicated that they were going back to their respective units and returned to their reserve status, whatever that was. Separation referred to when a soldier separated from the service because their pre-established contract of service was complete and they had less than 20 years of service.

Once enrolled, participants in the intervention arm were given standard care management plus the mCare application, with all of its aforementioned features, and followed for up to 36 weeks. As part of standard care management, the CBWTU case managers contacted rehabilitating Service Members weekly, via telephone, to schedule appointments with medical specialists (which were individualized, according to the Service Member’s needs), monitor appointment attendance, assess progress toward therapy goals, and provide emotional support. Also as part of standard care management, Platoon Sergeants contacted the Service Members daily, by telephone, to monitor and address any logistical obstacles the Service Member might have encountered and consulted with other CBWTU staff members as needed.

In addition to the daily mCare questionnaires, participants completed study-related questionnaires about their well-being (i.e., quality of life, symptoms) and goals at baseline and every four weeks until the study ended or they out processed. At the end of their participation, participants were also asked to complete the System Usability Scale (SUS). Study-related surveys were administered through the online system called Survey Monkey^®.

Additionally, participants were approached to complete semi-structured interviews about the usability of mCare. The interviewees were kept anonymous to the investigators by de-identifying the data in the report.

Measures

Usability. The study examined participants’ perceptions of usability with the SUS and semi-structured interviews.

The SUS was developed in 1986 and is a recognized industry standard.⁵ It is technology independent, has been widely used in health and medical research to assess the usability of hardware,⁶ consumer software,⁷ websites,⁸ and cell phone technologies,⁹ and provides consistent reliability even with small sample sizes.¹⁰ The SUS has 10 questions, with responses ranging from strongly disagree to strongly agree (coded from 1 to 5), and reversed coded as needed so that all items had consistent scoring. The ten numerical scores are summed and multiplied by 2.5 to obtain the overall SUS value, ranging from 0 to 100. In addition to the overall SUS score, the SUS has two sub-scores: usability and learnability.^11,12 Learnability is defined as a component of usability, and is “the degree to which a product or system can be used by specified users to achieve specified goals of learning to use the product or system with effectiveness, efficiency, freedom from risk and satisfaction in a specified context of use is the ease of which a user can learn how to software tools” (ISO/IEC 25010:2011, Section 4.2.4.2). The overall SUS score, usability sub-score, and learnability sub-score can be used to determine a percentile ranking and a corresponding conventional letter grade (with the values ‘A’, ‘B’, ‘C’, ‘D’, ‘F’), based on published baseline data for cell phone technologies.¹³ The letter grade indicates how usable mCare is scored relative to other cell-phone based tools and is straightforward to interpret.

The semi-structured interviews were conducted by the Sister Kenny Research Center (SKRC). The interviews took place over the telephone and were 10-15 minutes in duration. SKRC transcribed the interviews and analyzed the transcripts using qualitative analysis techniques. To assure quality of analysis, three team members coded three interviews from mCare, compared their results, discussed differences, and reached consensus on rule-based coding criteria. Valence was assigned to each response when applicable, representing the overall tone (i.e., positive vs. negative).

Behavioral Health, mTBI and PTS Status. Due to the high rates of these problems among Service Members returning from Iraq and Afghanistan, the study sought to determine whether having a behavioral health problem, mTBI, or PTS was related to perceptions of usability. In this cohort, the most common behavioral health problems are depression and anxiety. We grouped participants as follows: a) no behavioral health problem, mTBI, or PTS; b) behavioral health problem(s) only; c) behavioral health problem(s) plus mTBI and/or PTS; and d) mTBI and/or PTS only. Because mTBI is not a behavioral health diagnosis and many of its symptoms are similar to those for PTS, mTBI and PTS were grouped together.

Technological and Environmental Characteristics. The study collected data on several technology and environmental factors that could potentially affect the participants’ experience of mCare, namely the type of phone model they had (iPhone, Android, other), wireless carrier/service provider (AT&T, Sprint, T-Mobile, Verizon), and population size of the town in which the participant lived. For population size, the analysis grouped participant according to whether their town was a) at or below the median population size for the sample or b) above the median population size for the sample. Population size was considered a proxy for wireless service coverage.

Usage Measures. The mCare system automatically collects data on the date and time of when a questionnaire or an appointment reminder was sent successfully, if/when it synced with a participant’s mobile phone, and if/when the participant responded. Using this information, the study team created five measures to characterize usage: 1) the total number of questionnaires successfully sent to the participant; 2) the total number of appointment reminders successfully sent to the participant; 3) the percentage of questionnaires that synced with the participant’s mobile phone; 4) the percentage of appointment reminders that synced with the participant’s mobile phone; and 5) the percentage of successfully synced daily mCare questionnaires that the participant responded to. A ‘sync’ occurred when the user logged into the mCare application, which initiated an automatic information exchange that made current content available on the user’s phone. Responding to the questionnaires required more effort on the part of the participant than other features of mCare.

Other Background Characteristics. The analysis included age (years, >/= 18), gender, marital status (single, married, divorced), living arrangement (alone, with a spouse or partner, or ‘other’, meaning with friends or family), military rank (Enlisted, Warrant Officer, Officer), CBWTU site (in Alabama [AL], Virginia [VA], Illinois [IL], or Florida [FL]), and number of weeks enrolled in the study. Educational attainment was not collected in this study; however, a minimum of a high school diploma is required to enlist in the United States military.

Analysis

First, the analysis summarized the background, technological, environmental and usage characteristics of the sample and tested for differences between participants who completed the SUS and those who did not, using t-tests and Fisher’s Exact Test. Second, the analysis calculated the total SUS score and the usability and learnability sub-scores for the study participants overall (of those who responded to the SUS) and then by key participant characteristics. Background characteristics that differed between those who completed the SUS and those who did not were included in the usability assessment as well. The analyses tested for group differences in SUS scores using Analysis of Variance (ANOVA) or t-tests. Third, from the semi-structured interviews, the analysis examined the frequency (and percentage) of the valence (i.e., positive, negative, neutral) of each question pertinent to general mCare usage. Fourth, the analysis calculated Pearson correlations coefficients to determine the bivariate associations between usability and usage.

Ethics Statement

The study protocol was reviewed by the Institutional Review Board in the Department of Clinical Investigation at Walter Reed National Military Medical Center.

Results

The average age of the study participants in the mCare arm at time of enrollment was 38 years (Table 1). Most participants were male (84.2%), married (60%), living with a spouse or partner (61.1%), and had the rank of ‘Enlisted’ (87.4%). Almost 51% of participants had Android phones, and 35.8% had iPhones. AT&T was the more widely used service provider (37.9%), followed by Verizon (33.3%). The population size of the towns participants resided in ranged from 682 to 420,003 residents. In the intervention arm, 79% lived in or near towns below the median population size. Forty-two percent had no behavioral health problems, TBI, or PTS, 23.2% had a behavioral health problem only, and the remainder had mTBI and/or PTS with or without a behavioral health diagnosis also. Participants synced/retrieved 86.5% and 93.1% of daily questionnaires and appointment reminders sent during the active intervention, respectively, and responded to 60.3% of the questionnaires synced.

Table 1: Characteristics of the mCare Participants, Overall and Whether They Completed the SUS Questionnaire

Forty-one (43.2%) of the mCare participants completed the SUS. Those who completed the SUS differed from those who did not in that they were older (p = 0.02), synced/retrieved more appointment reminders (p = 0.06), and responded to more questionnaires (p = 0.003) on average. Also, higher percentages of participants who lived with a spouse/partner (p = 0.07), were of higher rank (p = 0.05), and lived in more populated towns (p < 0.0001) completed the SUS than did not.

The mCare application received high marks for the total SUS score. Mean total SUS score was 78 [standard deviation (SD) = 21.2], for a letter grade of ‘A’, mean usability sub-score was 76.7 (SD = 22.9), for a letter grade of ‘A’, and mean learnability sub-score was 82.0 (SD = 25.0), for a letter grade of ‘A+’ (Table 2). Participants with a behavioral health problem only gave mCare lower marks (‘C+’ for both total score and usability sub-score, and ‘B-’ for learnability sub-score), whereasparticipants with no behavioral health problem, mTBI, or PTS gave mCare an ‘A+’ for all domains. Although seemingly large, the differences in SUS scores by behavioral health, mTBI or PTS status were not statistically significant. Similarly, differences in SUS scores appeared to differ across the age groups and by rank, but the differences were not statistically significant. Participants living with a spouse or partner gave mCare ratings of ‘A+’ across all domains, whereas participants in ‘other’ living arrangements gave mCare a ‘C’ for the total scale and a ‘D’ for the usability sub-score (p = 0.06).

Table 2: SUS Scores by Selected Participant Characteristics

With respect to technology and environmental measures, participants who used wireless carriers other than AT&T rated mCare as an ‘A-’ or better on all domains. Participants with AT&T gave mCare a ‘B+’ for total the total score, a ‘B’ for the usability sub-score, and an ‘A+’ for learnability. For phone type, mCare again received letter grades of ‘A’ or better, except from iPhone users, who gave mCare a ‘B’ for total score and ‘C+’ for usability sub-score (p = 0.07).

Forty-nine of the 95 (51.6%) mCare participants completed semi-structured interviews. For all questions, the percentage of positive responses was greater than the percentage of negative and neutral responses combined. The question with the most favorable response (91.8% positive) was about how mCare facilitated communication with the care team (Table 3). The questions with the least favorable responses were about reporting pain (22.4% negative or neutral, with 85.7% of interviewees saying pain had been a concern during the study) and the Appointments/Reminder feature (22.4% negative or neutral). In response to the question about communication, one interviewee said: “I think it was a good program actually, the way you can put everything in your phone… One thing I didn’t like, the program is a little slow. If you want to load up five or six appointments at a time because you have a lot of appointments, it always seemed like it took a lot….” For the pain question, the greater negative/neutral responses were more about usefulness than usability per se. One interviewee said, “If I told you that at noon my pain was three and then at one pm it was seven and then at five pm it went back to three…. and it was awful when I woke up, what would you do about it?”

Table 3: Valence Ratings of mCare Participants’ Responses to Usability Questions in Semi-Structured Interviews

The correlation coefficients show that completion of more of the daily questionnaires was associated with higher total SUS scores (r = 0.46, p = 0.004), higher usability sub-scores (r = 0.44, p = 0.006), and higher learnability sub-scores (r = 0.36, p = 0.021) (Table 4).

Table 4: Correlations among Usability and Usage Measures

Discussion

Overall, results from the SUS and semi-structured interviews indicate that users generally rated the mCare application favorably. The heterogeneity of the cohort served in the mCare RCT warranted analysis of whether and how certain social groups rated the mCare application. This type of analysis is about the ‘fit’ between a person and the technology, and can help guide a provider in determining who would benefit from using a particular technology to support their self-care.

Previous research shows that information technologies, such as the Internet, phones, and social media, can improve patient adherence, information exchange between providers and patients, and outcomes.¹⁴ With respect to mobile phones specifically, one meta-analysis of 16 studies showed that mobile phone text messaging led to a significant increase in medication adherence,¹⁵ and a systematic review of seven interventions for cardiovascular disease management found mobile phone applications effective in increasing medication adherence and physical activity.¹⁶ Likewise, a pilot study of mCare found an increase in adherence to health care appointments.¹⁷

Many health care providers recognize that, despite how necessary self-care is, it is not necessarily the highest priority of their patients, and/or patients have competing priorities. The mCare team hypothesized that a more satisfying user experience would lead to consistent and sustained usage. Yet Sawesi and colleagues’ systematic review found that only about one-third of the 170 studies examined addressed usability.¹⁴ The results of this analysis support that hypothesis, showing that higher participant response to the daily questionnaires – the most intensive aspect of the mCare application – was statistically associated with higher ratings of usability. A previous analysis of mCare found that patient engagement was high (participants responded to over 60% of the daily questionnaires) throughout the 36 weeks of the intervention as well.¹ The present analysis, however, cannot clarify how much usability drives usage and how much usage drives perceptions of usability.

Moreover, this examination showed that ratings of usability were lower among participants who used an iPhone. Perceptions of usability according to phone type is important because the study was designed on a bring-your-own-device (BYOD) model, which is consistent with what is likely to happen when health care providers offer patients mobile applications such as mCare. The study team had to make mCare compatible with over 400 different phone models and available on all four major United States (US) cellular service providers in 2008. This meant designing mCare with the limitations of certain phone makes and models in mind, and because iPhone historically led the trend toward mobile applications and is known for its attention to user experience, it is not surprising that mCare did not meet their expectations.

Usability ratings were also lower among participants who used AT&T as their carrier (though not statistically significant). Although AT&T is the second largest United States network, at the time of the RCT, coverage maps from OpenSignal (http://opensignal.com/) show that the density of its tower locations is less than that of the other networks in the areas where the study participants were located, and its 3G download speed was slower than that of its competitors (e.g., 2.4 Mbps versus 4.7 Mbps for T-Mobile).¹⁸ This could have meant more difficulty for the user to sync with mCare to obtain the information being sent to them, particularly the links to the questionnaires. As of 2017, however, the situation is different, with convergence occurring in the user interface and functionality of mobile phones, despite brand, as well substantial growth in cellular coverage and download speeds.

Usability ratings were also lower among participants who lived alone or in a setting other than with a spouse/partner. A possible explanation could be the interaction of age and living arrangement. That is, although there were not statistically different usability ratings across age groups, perhaps the intersection of age and living arrangement is important. Study participants who did not live with a spouse/partner tended to be younger in this study (i.e., 42.9% of the participants who had ‘other’ living arrangement were in their 20s, compared with 0% of participants who lived alone and 19.0% of participants who lived with a spouse/partner), and study participants who lived alone tended to be older (40.0% of the participants who lived alone were older than 50 years, compared with 19.0% who lived with a spouse/partner and 28.6% who lived in ‘other’ arrangements). Research suggests the age groups differ in terms of their expectations of mobile phones and applications, with younger cohorts tending to have higher expectations.^19,20 Alternatively, it might have been the case that not having a significant other to assist with mCare affected perceptions of usability in this cohort of injured Service Members. The study design cannot address these questions directly because the number of participants in sub-groups is too small, so they are opportunities for future research.

The results showed that usability scores were lowest for participants who had a behavioral health problem. Behavioral health problems included anxiety and depression. A literature review spanning three decades concluded that patients with depression and/or anxiety are less likely to be compliant and therefore have poorer outcomes.²¹ Although the differences are not statistically significant, the variation in the usability scores could be the effect of patients with behavioral health problems seeing overall poor outcomes attributing these outcomes to the mCare system. By contrast, the results for participants with mTBI and/or PTS are consistent with a previous study showing that people who have had a trauma are receptive to the use of mobile tools for helping to track symptoms.²²

A limitation of this study is that 43.2 percent of the participants who received mCare completed the SUS at the time of their exit from the study. Participants in the standard care management group had a similar rate of missing for study questionnaires, which suggests that dissatisfaction with mCare was not necessarily the reason for non-response on the SUS. The analysis attempted to account for non-response on the SUS by testing for differences in background characteristics between those who completed the SUS versus those who did not, and then looking for associations between salient background characteristics and usability. The analysis found that those who completed the SUS were older and tended to be living with a spouse/partner. As noted above, living arrangement was associated with usability ratings as well. All other background characteristics were similar between those who completed the SUS and those who did not, suggesting that the observations about usability are generalizable to the rest of the study sample.

When the mCare study was launched, data plans were not as available and inexpensive as they are in 2017. On one hand, the cost of using mCare was potentially a concern among Service Members with limited economic resources, had limited data plans and/or could not afford overages, and not including these Service Members could have resulted in bias. On the other hand, offering high stipends can unintentionally result in a different kind of bias and also be coercive. The study tried to balance these issues by offsetting the costs of mCare for the participant by providing participants with a subsidy for their cellular service plans in the amount of $50 a month for unlimited data and voice, which was in alignment with the cost of that level of service on all service providers. The current cellular service market allows for much higher levels of cellular usage and smart phone adoption than at this study was conducted.

Conclusion

This evaluation of the mCare application for injured Service Members rehabilitating in their communities found it to be usable and easy to learn. Further, relative to the large database of SUS scores for mobile tools that was used to develop the letter grades scheme for interpreting individual scores, mCare tends to score high marks. Type of phone, living arrangement, and amount of usage of the application relate to usability scores. Although not statistically significant, usability was rated lower by people with behavioral health problems, those who were older, and/or had AT&T as their carrier. These things should be considered in determining patients who would benefit from using mCare in the future.

Acknowledgements

This study was funded by the Office of Force Health Protection, and managed by the Mobile Health Innovation Center (MHIC), an intramural laboratory of the Telemedicine and Advanced Technology Research Center (TATRC), United States Army Medical Research and Material Command (USAMRMC).

The authors would like to thank the following individuals for their contributions towards conducting the mCare clinical research study: Mike Bairas, Joyce Bennett, Charles Blair, Mabel Cooper, Eric Cutlip, COL (Ret) Karl Friedl, Ryan Fung, Lois Goldstein, Cindy Gilley, Willie Ho, Enouch Hui, Kurt Huttar, Jonathan Kean , Marsha Ma, Amanda Martin, Mahvish Malik, COL (Ret) Eileen McGonagle, COL (Ret) Fran McVeigh, COL (Ret) Ron Poropatich, April Pradier, Mary Radomski Celicia Thomas, COL (Ret) Johnnie Tillman, Matthew White, Rick Wise, Ping Xiang, Sarmid Youhanna, and Edmound Yu. Additionally, the authors would like to express their gratitude to the staff at the Community Based Warrior in Transition Units (CBWTUs) in Alabama, Florida, Illinois and Virginia where the study took place.

The views and opinions expressed in this articles are those of the authors and do not reflect official policy or position of the Department of the Army, Department of Defense or the U.S. Government.

Declaration of Competing Interests

All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: HP, MC, LG, JT and SF had financial support from Geneva Foundation (recipient of a a grant from the Office of Force Health Protection, and managed by the Mobile Health Innovation Center, an intramural laboratory of the Telemedicine and Advanced Technology Research Center, United States Army Medical Research and Material Command) for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.

References

1. Pavliscsak H, Little J, Poropatich R, et al. Assessment of Patient Engagement with a Mobile Application among Service Members in Transition. J Am Med ­Inform Assoc 2016;23:110–8.

2. Schnall R, Rojas M, Bakken S, et al. A user-centered model for designing consumer mobile health applications (apps). J Biomed Inform 2016 Feb 20. pii: S1532-0464(16)00024-1. doi: 10.1016/j.jbi.2016.02.002. [Epub ahead of print].

3. Wicklund ME, Wilcox SB. Designing Usability into Medical Products. Boca Raton, FL: CRC Press 2005.

4. Fonda SJ, Paulsen CA, Perkins J, et al. Usability test of an Internet-based informatics tool for diabetes care providers: the Comprehensive Diabetes Management Program. Diabetes Technol Ther 2008;10:16–24.

5. Brooke J. SUS: a “quick and dirty” usability scale. In: PW Jordan, B Thomas, BA Weerdmeester, AL ­McClelland, eds. Usability Evaluation in Industry. London: Taylor and Francis, 1996: 189–194.

6. Randell R, Backhouse MR, Nelson EA. Videoconferencing for site initiations in clinical studies: Mixed methods evaluation of usability, acceptability, and impact on recruitment. Inform Health Soc Care 2015;22:1–11.

7. Sousa VE, Lopes MV, Ferreira GL, et al. The construction and evaluation of new educational software for nursing diagnoses: a randomized controlled trial. Nurse Educ Today 2016;36:221–9.

8. Starling R, Nodulman JA, Kong AS, et al. Usability testing of an HPV information website for parents and adolescents. Online J Commun Media Technol 2015;5:184–203.

9. Kalz M, Lenssen N, Felzen M, et al. Smartphone apps for cardiopulmonary resuscitation training and real incident support: a mixed-methods evaluation study. J Med Internet Res 2014;19:3.

10. Sauro J. Measuring Usability with the System ­Usability Scale (SUS). February 2, 2011. http://www.measuringu.com/sus.php. Accessed March 15, 2016.

11. Lewis JR, Sauro J. The factor structure of the system usability scale. International conference (HCII 2009), San Diego, CA, USA.

12. Borsci S, Fedrerici S, Lauriola M. On the dimensionality of the System Usability Scale: a test of alternative measurement models. Cognitive Processing 2009;10:193–197.

13. Bangor A, Kortum PT, Miller JT. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009;4:114–123.

14. Sawesi S, Rashrash M, Phalakornkule K, et al. The impact of information technology on patient engagement and health behavior change: a systematic review of the literature. JMIR Med Inform 2016;41:e1.

15. Thakkar J, Kurup R, Laba TL, et al. Mobile telephone text messaging for medication adherence in chronic disease: a meta-analysis. JAMA Intern Med 2016 Feb 1. doi: 10.1001/jamainternmed.2015.7667. [Epub ahead of print]

16. Pfaeffli Dale L, Dobson R, Whittaker R, et al. The effectiveness of mobile-health behaviour change interventions for cardiovascular disease self-management: A systematic review. Eur J Prev Cardiol 2015 Oct 21. pii: 2047487315613462. [Epub ahead of print]

17. Poropatich RK, Pavliscsak HH, Tong JC, et al. mCare: using secure mobile technology to support soldier reintegration and rehabilitation. Telemed J E Health 2014;20:563–9.

18. OpenSignal.com. http://opensignal.com/network-coverage-maps/; Accessed 25February2016.

19. Archana Kumar, Heejin Lim. Age differences in mobile service perceptions: comparison of Generation Y and baby boomers. Journal of Services Marketing 2008;22:568–577.

20. Mark MH, Goode FD, Luiz M, et al. Determining customer satisfaction from mobile phones: a neural network approach. Journal of Marketing Management 2005;21:755–778.

21. DiMatteo MR1, Lepper HS, Croghan TW. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern Med 2000;160:2101–7.

22. Price M, Sawyer T, Harris M, et al. Usability evaluation of a mobile monitoring system to assess symptoms after a traumatic injury: a mixed-methods study. JMIR Ment Health 2016;3:e3.

Read More