Pages Menu
TwitterRssFacebook

Posted on Jul 16, 2019 in Original Article | 0 comments

Myanmar is ready to engage mHealth applications for improved postoperative care

 

Sariah Khormaee MD PhD1, Athena Nguyen2, Esther Bartlett3, Michael Lwin4, Peter Chang MD5, Misja Ilcisin6

1Hospital for Special Surgery, Orthopedic Surgery

2Santa Clara University College of Arts and Sciences

3Santa Clara University College of Arts and Sciences

4KoeKoe Tech Co., Ltd

5Washington University, Orthopaedic Surgery

6KoeKoe Tech Co., Ltd

Corresponding author: sariah.khormaee@gmail.com

Journal MTM 8:1:29–36, 2019

doi:10.7309/jmtm.8.1.4


Background: There is incredible potential for telemedicine to advance postoperative care. Work in high-income nations shows the potential to use mobile phones to monitor postoperative recovery progress. However, there is little information about the attitudes of people in low resource countries, like Myanmar, toward the adoption of mHealth in postoperative care.

Aims: This study presents survey results collected in Myanmar to better understand cultural attitudes of this population towards adopting mHealth technologies to improve postoperative patient care.

Methods: A thirteen-question survey was developed, focused on demographic questions and attitudes towards physicians, the internet, and willingness to perform tasks on their mobile phones. Respondents were selected in a sample of convenience in urban and rural public spaces.

Results: Of the 125 people approached, 112 agreed to participate in the survey. A wide range of ages (18-78), genders (55.4% female), locations (22.3% rural, 77.7% urban) and ethnicities (67% Burmese) were represented. 85.7% were willing to make contact with a surgeon in a hypothetical postoperative setting via mobile phone. 83.0% were willing to fill out a survey about their postoperative state and 69.6% were willing to send a picture of their wound with their surgeon via mobile phone. A majority of respondents had a very high level of trust in physicians in general, most already owned a mobile phone with access to the internet and used it to look up health information.

Conclusion: Our results indicate that Myanmar could provide a promising location for the implementation of mHealth technologies to improve post-operative care.

Keywords: mobile health, telecare, health information on the Web, ehealth, assistive technologies


Read More

Posted on Jul 16, 2019 in Original Article | 0 comments

mHealth Can Activate Patients to Discuss Hepatitis C Screening with Physicians

 

John B. Kellogg, BA1, Jessica D. Lee2, Daniel R. Murphy, MD, MBA1,3, Monisha Arya, MD, MPH1,3

1Baylor College of Medicine, One Baylor Plaza, Houston, TX 77030, USA;

2Rice University, 6100 Main St, Houston, TX 77005, USA;

3Center for Innovation in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center, 2002 Holcombe Blvd, Houston, TX 77030, USA;

Corresponding Author: Kellogg@bcm.edu

Journal MTM 8:1:20–28, 2019

doi:10.7309/jmtm.8.1.3


Background: Despite the prevalence of hepatitis C virus (HCV) and the availability of effective treatments, HCV screening remains suboptimal, in part due to primary care physicians’ (PCPs) unawareness of and discomfort discussing HCV risk factors. Patient-facing text message campaigns may overcome these barriers by empowering patients to initiate screening discussions with their PCPs.

Aims: The objectives were to evaluate a patient-facing text message campaign in terms of (1) feasibility, (2) acceptability, and (3) impact on patient-PCP discussions about HCV screening.

Methods: Primary care patients were recruited to receive either an HCV text message, which contained HCV information and a prompt to discuss HCV with their PCPs, or a calcium control text message. Forty minutes before their appointments, participants were sent their assigned text message. Participants were then called for an evaluation of the text message campaign.

Results: Of 185 patients called, 38 enrolled and completed the study. Participants who were sent an HCV text message (n=25) were significantly more likely to initiate a conversation with their PCPs about HCV screening than participants sent a calcium control text message (n=13) (p=0.008). Thirty-two (82%) participants liked receiving a health-related text message (88% in the HCV group; 70% in the control group).

Conclusions: A patient-facing HCV text message campaign shows promise as a novel method to activate primary care patients to initiate HCV screening discussions with their PCPs. This campaign may help educate patients about the importance of HCV screening, overcome physician barriers to screening, and, ultimately, help control the HCV epidemic.

Keywords: hepatitis C, text messaging, physician-patient relations, preventive health services, primary health care


Read More

Posted on Jul 16, 2019 in Original Article | 0 comments

Teleophthalmology through handheld mobile devices: a pilot study in rural Nepal

 

Karen Hong1, Sean Collon2, David Chang3, Sunil Thakalli4, John Welling4, Matthew Oliva4, Esteban Peralta5, Reeta Gurung6, Sanduk Ruit6, Geoffrey Tabin1,4, David Myung1,7, Suman Thapa6

1Byers Eye Institute, Stanford University School of Medicine

2Vanderbilt University School of Medicine

3Los Altos Eye Physicians, Los Alto, CA

4Himalayan Cataract Project

5University of Florida School of Medicine

6Tilganga Institute of Ophthalmology, Kathmandu, Nepal

7Division of Ophthalmology, Veterans Affairs Palo Alto Health Care System, Palo Alto, CA

Corresponding Author: suman.thapa@tilganga.org

Journal MTM 8:1:1–10, 2019

doi:10.7309/jmtm.8.1.1


Background: To compare screening referral recommendations made by remotely located ophthalmic technicians with those of an ophthalmologist examining digital photos obtained by a portable ophthalmic camera system powered by an iOS handheld mobile device (iPod Touch).

Methods: Dilated screening eye exams were performed by ophthalmic technicians in four remote districts of Nepal. Anterior and posterior segment photographs captured with a Paxos Scope ophthalmic camera system attached to an iPod Touch 6th generation device were uploaded to a secure cloud database for review by an ophthalmologist in Kathmandu. The ophthalmic technicians’ referral decisions based on slit-lamp exam were compared to the ophthalmologist’s recommendation based on the transmitted images.

Results: Using the transmitted images, the ophthalmologist recommended referral for an additional 20% of the 346 total subjects screened who would not have been referred by the ophthalmic technician. Of those subjects, 34% were referred to the retina clinic. Conversely, among the 101 patients referred by the technician, the ophthalmologist concurred with the appropriateness of referral in more than 97% of cases but thought eight (2.8%) of those patients had variants of normal eye pathology.

Conclusion: An ophthalmologist who reviewed data and photos gathered with the mobile device teleophthalmology system identified a significant number of patients whose need for referral was not identified by the screening technician. Posterior segment pathology was most frequently found by the remote reader and not by the technician performing dilated slit lamp examinations. These results are promising for further clinical implementation of handheld mobile devices as tools for teleophthalmic screening in resource-limited settings.

Keywords: telemedicine, rural population, ophthalmology, referral and consultation, global health


Read More

Posted on Dec 4, 2018 in Articles, Review | 0 comments

Professional Involvement in Ophthalmology iPhone Application Development: An Update

Professional Involvement in Ophthalmology iPhone Application Development: An Update

Louis Stevenson, MBBS1

1 The Royal Victorian Eye and Ear Hospital, East Melbourne, Victoria, Australia 3002

The author is not the recipient of a research scholarship.

Corresponding author: Louis.Stevenson@eyeandear.org.au

Journal MTM 7:2:1–6, 2018

doi:10.7309/jmtm.7.2.1


Background: Smartphone technology and related applications are increasingly prevalent in the field of medicine and ophthalmology, offering a wide range of hand-held capabilities not previously available. While these technologies have enormous potential, many apps are developed without the involvement of qualified professionals leading to concerns about their quality and validity.

Aims: To assess iPhone® applications aimed at eye care professionals for qualified professional involvement in their development.

Methods: Applications were identified by searching the Apple® (Cupertino, CA) iTunes® Store using the terms ‘ophthalmology’ and ‘ophthalmologist’ in addition to a number of common eye conditions outlined by the Centers for Disease Control and Prevention. Applications were then assessed for category of application, intended audience, documented involvement of medical professionals in application development, price, user rating and date of publication.

Results: In total, 152 applications were identified across 12 categories. Applications were found to target eye-care professionals (ophthalmologists and non-ophthalmologists) (32.3%), ophthalmologists specifically (32.3%), non-ophthalmology eye-care professionals (3.3%) and patients (34.2%). Overall, 36 (23.7%) applications had clearly documented professional involvement in their development.

Conclusions: There continues to be a low level of professional involvement in the development of ophthalmology based iPhone® applications. This is concerning given the growing prevalence of these technologies and their enormous potential. It is therefore incumbent on clinicians to be informed about the applications they use and promote high quality applications developed with professional expertise.

Disclosures: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.

Keywords: ‘Ophthalmology’, ‘smartphone’, ‘technology’, ‘ophthalmologist’, ‘telemedicine’.


Introduction

Smartphones are mobile, handheld devices, with functional capabilities similar to those of laptop computers.1 These devices represent a significant technological milestone and provide users with a diverse range of easily accessible, handheld capabilities not previously available.

The field of medicine has increasingly been subject to the influence of smartphones and their applications (apps). Between 2001 and 2013, smartphone use among health professionals rose from 30%2 to 86%3, highlighting the dramatic uptake of this technology. Accompanying the rise in smartphone use has been a similarly dramatic increase in the number of health related apps.4, 5 The potential benefits of the technology are extensive and range from increased patient compliance, data management, displacement of old expensive technologies and increased communication capabilities.2

The specialty of ophthalmology has also been impacted by the development of smartphones, with a diverse range of ophthalmology themed apps now available.6 These apps target a wide range of audiences from ophthalmologists and non-ophthalmology eye care professionals, through to medical researchers and healthcare consumers.6, 7 The range of functions typically performed by these apps fall into several broad categories including; clinical examination and assessment tools, medical administration, professional and patient education, and clinical calculators.2, 58 The areas of telemedicine and teleophthalmology have also been impacted by smartphones9 with a Brazilian study finding smartphone based photography to be both sensitive and specific in the diagnosis of emergency eye conditions when used as a teleconsultation tool.10 These results are highly promising and have the potential to improve access to healthcare in isolated populations.

While these capabilities are unprecedented and offer enormous potential, there is concern that their utility is undermined by low quality app development and a lack of evidence supporting their use.2 A 2014 report by Cheng et al. found that less than one third of ophthalmology iPhone® apps available on the Apple® iTunes® Store, had documented medical input in their development.7

Such concerns are not limited to ophthalmology with similar findings having been made across other disciplines. A 2011 review of smoking cessation apps found many apps deviated significantly from relevant clinical guidelines11, while a 2016 review of health related smartphone apps found that the majority of apps targeting clinical practice lacked scientific evidence underpinning their use.12 Furthermore, apps are difficult to regulate and a number of privacy issues related to their use have been raised.13

Given these findings, a major challenge now facing clinicians, researchers, and consumers is harnessing the potential of this technology while avoiding its shortcomings. This report provides an update on the quality of ophthalmology apps available on the Apple® iTunes® Store with specific attention paid to the involvement of medical professionals in app development. These findings will be compared to those previously published to determine whether there has been a change in the quality of app development over recent years. This is a topic of significance given the recognised importance of evidence-based practice, and the potential for poor quality, or unproven technologies to undermine this.14

Materials and methods

The Apple® iTunes® Store was searched on Saturday 17 March 2018 to identify ophthalmology related apps. Where appropriate, this study has adopted some of the methodology used previously by Cheng et al.7 in order to facilitate comparisons between the two reports.

Inclusion criteria

Apps were considered for inclusion provided they were returned using the search terms ‘ophthalmology,’ and ‘ophthalmologist’. Additionally, the Apple iTunes Store was searched using terms derived from common eye disorders as outlined by the Centers for Disease Control and Prevention15 including ‘refractive’, ‘macular degeneration’, ‘cataract’, ‘diabetic retinopathy’, ‘glaucoma’, ‘amblyopia’, and ‘strabismus.’

Exclusion criteria

Apps relating to fields other than human ophthalmology such as veterinary ophthalmology or the physics of light were excluded. Apps were also excluded where the primary purpose was to promote a particular practitioner, conference, institution or product. Similarly, ‘demo’ or ‘lite’ versions of apps were excluded.

Data collection and statistical analysis

The following data were collected within the Apple® iTunes® Store based on information provided in the app product description; app name, developer, target audience, description of app function, category of app, cost, iTunes® rating, documentation of medical involvement in app development, year first published or copyright year, and size of the app in megabytes (MB). Professional involvement was defined as clear documentation of medical professional involvement or reputable institutional involvement such as that by a university or hospital in app development. Individual medical practitioners did not need to be identified in order to satisfy this criterion. Information was not sought from external websites, via contacting app developers or within the apps themselves. The data were analysed using descriptive statistics in Excel (Microsoft, Redmond WA).

Ethics

This research was conducted in accordance with the relevant ethical guidelines. No human participants were involved.

Results

A total of 152 ophthalmology themed apps were identified with the above search terms. Apps were categorised into 11 categories according to their primary function in addition to a miscellaneous category (Fig. 1).

jmtm.7.2.1f1.jpg

Figure 1: Categories of ophthalmology applications and their relative distributions

Eleven apps (7.2%) were aimed at providing educational material to all eye-care professionals (ophthalmologists and non-ophthalmologists) and included atlases and written reference material. Five (3.3%) apps were aimed at providing educational material to non-ophthalmology healthcare professionals and included flash cards and quizzes. Eleven (7.2%) apps provided educational material aimed at ophthalmologists specifically and primarily included written reference material. Six (3.9%) apps were electronic versions of academic journals. Eleven (7.2%) apps provided patient education including information delivered in written and multimedia formats. Furthermore, a number of apps provided ophthalmologists with images to be used for patient education such as in the setting of pre-operative counselling. Thirty-seven (24.3%) apps were clinical examination tools and included visual acuity charts, Amsler grids and colour vision testing plates. Twenty (13.2%) apps were clinical calculators of various types including toric and non-toric intraocular lens calculators, glaucoma risk calculators, and visual acuity converters.

Twenty-one (13.8%) apps provided treatment for amblyopia and strabismus through games or visual tasks. Seven (4.6%) apps functioned as low vision aids by providing screen magnification or allowing users to change the display colour scheme to better suit those with colour vision deficits. Five (3.3%) apps were aimed at medication compliance, all of which were eye drop reminders. Sixteen (10.5%) apps were categorised under miscellaneous. Apps in this category included an operator simulator, ophthalmology media, clinical administration, social media and telehealth.

The apps identified on the iTunes® Store were aimed at 4 different audiences. Forty-nine (32.3%) apps were targeted at ophthalmologists, while 5 (3.3%) and 46 (30.3%) were targeted at non-ophthalmology eye care professionals and eye care professionals (ophthalmologists and non-ophthalmologists), respectively. Fifty-two (34.2%) apps were targeted at patients.

Ratings

Forty-two (27.6%) apps had five or more user ratings resulting in them having an overall rating on the iTunes® Store. The mean rating amongst these apps was 3.4 stars. In contrast, 110 (72.4%) apps did not have an average user rating provided as they had been rated less than 5 times.

Medical professional involvement

In total, 36 (23.7%) apps had clearly documented medical involvement, or were developed by a reputable organisation such as a hospital or university. There were 34 (22.4%) apps that were assumed to have medical or professional input into their development because of their content or presentation, however they did not explicitly state this. A total of 82 (53.6%) apps had no professional involvement in their app development (Tab. 1).

Table 1: Professional involvement in app development by category of app

Year of publication

The number of apps produced each year from 2009 to 2018 (inclusive) was; 3 (2.0%), 8 (5.3%), 15 (9.9%), 10 (6.6%), 15 (9.9%), 18 (11.8%), 29 (19.1%), 20 (13.2%), 23 (15.1%) and 11 (7.2%) (Fig. 2). The 2018 figure of 11 represents the number of apps produced until 17 March 2018 when the search was conducted.

jmtm.7.2.1f2.jpg

Figure 2: Number of ophthalmology themed apps by year of publication or copyright

Price

The mean and median price of apps was USD$5.25 and $USD0.00, respectively. The price of apps ranged from USD$0.00 – USD$99.99. Eighty-nine (58.6%) apps were free while 8 (5.3%), 11 (7.2%), 7 (4.6%), 19 (12.5%) and 19 (12.5%) apps were USD$0.99, USD$1.99, USD$2.99, USD$3.00 – USD$10.00 and >USD$10.00 respectively.

Discussion

This report provides an update on the involvement of medical professionals in the development of ophthalmology apps available through the Apple® iTunes® Store. A similar report published in 2014 identified a total of 182 ophthalmology themed apps compared to 152 identified in this report,7 however significantly broader search terms were used in this previous study. As such, these results likely confirm that there are an increasing number of ophthalmology apps available. This is supported by the significant increase in number of ophthalmology apps released each year. While three apps were released in all of 2009, 11 apps have already been released as of 17 March 2018 highlighting the rapid growth in ophthalmology themed iPhone® apps over the past decade.

A broader number of app categories were identified in this report compared to those identified in previous studies.7 Examples of these new categories include low vision aids, those targeting medication compliance in the form of eye drop reminders and apps that provide treatment for strabismus and amblyopia.

Clearly documented professional involvement or clear documentation of reputable institution involvement in app development was low, with only 23.7% of apps fulfilling these criteria. These findings are considerably lower than figures quoted in previous studies.7 This is especially concerning given the very low standard required to meet these criteria. For example, merely stating that there was doctor involvement was sufficient with no requirement to provide evidence for these claims. Additionally, simply having medical involvement in app development does not in any way validate an app against non-iPhone® standards. It is not within the scope of this report to assess the validity of all available apps however. When combined with those apps that were assumed to have professional involvement this figure rose to 46.1%.

While a lack of professional involvement in some categories such as eye drop alarms may be benign, in other categories of apps it may be dangerous. For example, almost 86% of apps that aimed to provide ‘lazy eye training’ in the setting of amblyopia and strabismus had no professional involvement in their development. These apps primarily consisted of games and superficially they may appear harmless, however they may represent an opportunity cost and deny patients of time spent performing proven treatments. This highlights the need to validate these apps to avoid adverse outcomes. Additionally, apps that provide visual assessment tools such as visual acuity or colour vision testing may provide clinicians and patients with false information if improperly designed.

Such limitations have already been identified within the literature. A 2015 report by Perera et al. investigating the validity of iPhone® based visual acuity charts failed to identify a single app of sufficient accuracy for clinical use16 while discrepancies between iPhone® based, and standard Ishihara charts have also been documented.17 These findings are particularly concerning given that clinical examination apps account for approximately one-quarter of the app market and have the potential to directly lead to clinical errors and adverse outcomes.

While the large number of free apps available may seem beneficial, it may have the effect of promoting the development of poor quality apps due to lower standards expected and tolerated by users. Additionally, the large number of apps aimed at non-ophthalmologists may also promote poor development given that this target audience is likely to lack the expertise needed to critically appraise the technology.

These findings are a significant concern given the growing prevalence and influence of smartphone technology in medicine and ophthalmology.25 With the exception of a few, current smartphone apps are poorly designed, lack evidence to support their use and as such, the technology is not reaching its full clinical potential. Despite this, it seems highly unlikely that ophthalmology based smartphone technology will not continue to grow and offer an ever increasing range of functions. As such, it is incumbent on clinicians to promote high-quality, evidence-based apps where possible in order to derive maximum benefit from this technology, whilst avoiding the inherent pitfalls.

Conclusion

This report provides and an up to date review on the quality of ophthalmology iPhone® app development with the results highlighting the low level of professional involvement in this process. This is concerning given the growing prevalence of these technologies and their enormous potential. It is therefore incumbent on clinicians to be informed about the applications they use and promote high quality applications developed with professional expertise.

References

1. BinDhim NF, Freeman B, Trevena L. Pro-smoking apps for smartphones: the latest vehicle for the tobacco industry? British Medical Journal. 2012;23(1):e4[1–8].

2. Mosa AS, Yoo I, Sheets L. A systematic review of healthcare applications for smart phones. BMC Med Inform Decis Mak. 2012;12(67).

3. Epocrates. 2013 mobile trends report [Available from: http://www.epocrates.com/oldsite/statistics/2013 Epocrates Mobile Trends Report_FINAL.pdf.

4. Kulendran M, Lim M, Laws G, Chow A, Nehme J, Darzi A, et al. Surgical Smartphone Applications Across Different Platforms: Their Evolution, Uses, and Users. Surgical Innovation. 2014;21(4):427–40.

5. BinDhim NF, Trevena L. There’s an App for That: A Guide for Healthcare Practitioners and Researchers on Smartphone Technology. Online Journal of Public Health Informatics. 2015;7(2):e218–5.

6. Zvornicanin E, Zvornicanin J, Hadziefendic B. The Use of Smart Phones in Ophthalmology. Acta Inform Med. 2014;22(3):206–9.

7. Cheng NM, Chakrabarti R, Kam JK. iPhone applications for eye care professionals: a review of current capabilities and concerns. Telemed J E Health. 2014;20(4):385–7.

8. Bastawrous A, Cheeseman RC, Kumar A. iPhones for eye surgeons. Eye. 2012;26:343–54.

9. Grisolia ABD, Abalem MF, Lu Y, Aoki L, Matayoshi S. Teleophthalmology: where are we now? Arq Bras Oftalmol. 2017;80(6):401–5.

10. Ribeiro AG, Rodrigues RA, Guerreiro AM, Regatieri CV. A teleophthalmology system for the diagnosis of ocular urgency in remote areas of Brazil. Arq Bras Oftalmol. 2014;77(4):214–8.

11. Abroms LC, Padmanabhan N, Thaweethai L, Phillips T. iPhone Apps for Smoking Cessation: A Content Analysis. American Journal of Preventative Medicine. 2010;40:279–85.

12. Gan SK-E, Koshy C, Nguyen P-V, Haw Y-X. An overview of clinically and healthcare related apps in Google and Apple app stores: connecting patients, drugs, and clinicians. Scientific Phone Apps and Mobile Devices. 2016;2(8).

13. BinDhim NF, Trevena L. Health-related smartphone apps: regulations, safety, privacy and quality. BMJ Innovations. 2015;1(2).

14. Birbeck GL, Wiysonge CS, Mills EJ, Frenk JJ, Zhou X-N, Jha P. Global health: the importance of evidence-based medicine. BMC Medicine. 2013;11(223).

15. Centers for Disease Control and Prevention. Common Eye Disorders United States of America: CDC; 2015 [Available from: http://www.cdc.gov/visionhealth/basics/ced/index.html.

16. Perera C, Chakrabarti R, Islam FMA, Crowston J. The Eye Phone Study: reliability and accuracy of assessing Snellen visual acuity using smartphone technology. Eye (Lond). 2015;29(7):888–94.

17. Sorkin N, Rosenblatt A, Cohen E, Ohana O, Stolovitch C, Dotan G. Comparison of Ishihara Booklet with Color Vision Smartphone Applications. Optometry and Vision Science. 2016;93(7):667–72.

Read More

Posted on Dec 4, 2018 in Original Article | 0 comments

Designing a WIC App to Improve Health Behaviors: A Latent Class Analysis

Designing a WIC App to Improve Health Behaviors: A Latent Class Analysis

Sylvia H. Crixell PhD, RD1, Brittany Reese Markides MS, RD2, Lesli Biediger-Friedman PhD, MPH, RD3, Amanda Reat MS, RD2, Nicholas Bishop PhD4

1Nutrition and Foods Professor, School of Family and Consumer Sciences, Texas State University, San Marcos, Texas; 2Nutrition and Foods Lecturer, School of Family and Consumer Sciences, Texas State University, San Marcos, Texas; 3Nutrition and Foods Assistant Professor, School of Family and Consumer Sciences, Texas State University, San Marcos, Texas; 4Family and Child Development Assistant Professor, School of Family and Consumer Sciences, Texas State University, San Marcos, Texas

Corresponding Author: scrixell@txstate.edu

Journal MTM 7:2:7–16, 2018

doi:10.7309/jmtm.7.2.2


Background: Smartphone apps have potential to effectively deliver health education and improve health behaviors among at-risk populations. To be successful, apps should include user input during stages of development. Previously, a prototype app designed for participants in the Texas Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) was developed based on input from focus groups.

Aims: This research aimed to continue app design by soliciting user input via a survey from a state-wide sample.

Methods: Texas WIC clients were asked about physical activity, healthy eating, and breastfeeding behaviors, stage of change regarding health behaviors, current use of health-related apps, and perceptions of app prototype features. Latent class analysis (n=942) was used to identify mutually exclusive groups based on the strength of participants’ agreement that prototype features would help them exercise more or consume more fruits and vegetables. Logistic regression examined health-related characteristics and sociodemographic differences between classes.

Results: Response to app prototype features was positive. A 2-class model best described latent classes. Class members that strongly agreed that prototype features would help them improve health behaviors were younger (< 35 years), not pregnant, already using health-related apps, and in the contemplation, preparation, or action stages of change regarding physical activity.

Conclusion: Refinement of the Texas WIC app should incorporate input from individuals who are pregnant, older than 35 years, or in pre-contemplation regarding physical activity. The iterative process of user-centered design applied in this research may serve as a useful framework for development of other public health apps.

Keywords: health promotion, technology, vegetables, smartphone, exercise


Introduction

In the United States, poverty affects women and children disproportionately, as they make up approximately 70% of the low-income population.1 Poverty is associated with deleterious health behaviors, such as consuming a low-quality diet and being physically inactive, particularly among vulnerable populations such as women and children.2 These behaviors contribute to serious health concerns, including poor birth outcomes, obesity, heart disease, type 2 diabetes, and certain cancers.2,3 Limited access to evidence-based information related to health, physical activity, nutrition, and infant care is a likely contributor to poor health behaviors and outcomes among low-income individuals and may be an important barrier that contributes to ongoing health disparities.2

Launched in 1972, the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) is a federal grant program that serves low-income pregnant, postpartum, and breastfeeding women, infants, and children up to age 5 who are at nutritional risk, with the goal of improving health behaviors and outcomes during critical periods of development.4 Annually, 8 million women, infants, and children are enrolled in WIC, with approximately 886 thousand participating in the state of Texas.5 WIC provides a number of resources, including vouchers for healthful foods to support pregnancy, lactation, and growth, and referrals to health care services.6 While they are enrolled in the program, WIC clients are expected to regularly participate in education that focuses on promoting healthy behaviors such as breastfeeding, exercise, and healthy eating (e.g. eating fruits and vegetables, cooking meals at home, eating meals as a family).6 Historically, WIC clinics have worked to impart evidence-based information related to health behaviors through education offered at clinics via face-to-face education. This education modality presents barriers to an already taxed population, which may lack reliable transportation to clinics and childcare during education sessions.7 In an attempt to mitigate these barriers, many WIC state agencies now offer online client education.8 However, reliable access to a computer with internet connectivity is not ubiquitous among Americans, and low-income smartphone owners are more likely to rely on their smartphones as a primary way of connecting to the internet.9 Indeed, Texas WIC clients have expressed a desire to receive education and services delivered via phone.7,10 Thus, smartphone apps may offer a viable alternative interface for providing innovative, accessible, and customizable health education to the WIC population. Research has supported the use of smartphones as a behavioral modification tool, with a number of applications developed to improve diet and physical activity.11

Smartphone apps have unique characteristics that may make them particularly ideal for delivering health education and supporting health behavior change. For example, smartphone apps can be used to access clients in real time, offer continual assessment of identified treatment goals, and deliver meaningful information and support to reinforce behavior change.11 Despite the promise of smartphone apps, individuals tend to discontinue app use after three months of downloading.12 Therefore, app developers should consider, a priori, the expressed needs of intended users. Indeed, all approaches to developing technological tools to improve health outcomes should engage people first.13 One approach to developing apps that prioritizes individuals is user-centered design (UCD), an evidence-based, iterative process prioritizing user input and engagement in designing products and services.14 Because UCD has been previously used to develop appealing smartphone apps that target health behaviors, such as physical activity,15 this process shows promise for developing an effective app for WIC clients. In 2014, based on strong interest among Texas WIC clients for delivery of nutrition education and services via their smartphones,16 the Texas Department of State Health Services WIC program commissioned us to develop a smartphone app prototype. To do this, we began the UCD process by conducting focus groups with a diverse sample of female WIC participants in south central Texas to explore current smartphone app use and preferences.17 Based on this initial user-input and tenets of the Social Cognitive Theory,18 we developed an app prototype, designed to provide customizable, interactive, and user-centered health education to the Texas WIC population.17 The prototype included features to support physical activity (i.e., activity calendar, activity tracker, exercise videos, resource library), healthy eating (i.e., meal calendar, healthy eating tracker, cooking videos, resource library, shopping list, fruit and vegetable game, farmer’s market locator), and breastfeeding (i.e., breastfeeding timer, growth chart, live assistant, resource library).17

The aim of this research was to continue the UCD process of developing an app for Texas WIC clients by disseminating a statewide survey seeking input regarding the app prototype features. Analysis of clients’ perceptions of prototype features designed to support physical activity and healthy eating are included in this report. Our approach was to use latent class analysis to identify subgroups of respondents based on the extent to which they agreed that the features would help them increase physical activity and intake of fruits and vegetables, and logistic regression to examine how membership in the latent classes were associated with sociodemographic and health-related characteristics. Recommendations for continued user-centered design of the WIC app were informed by characteristics of respondents in latent classes.

Methods

Sample

The survey was posted on the Texas WIC website from September 9, 2014 through November 6, 2014. Clients visiting the website were greeted with a pop-up window presenting an offer to take the survey in English or Spanish. Additionally, clinics in central Texas who had access to client email addresses sent invitations to clients to take the survey. Of the 606 emails sent, 88 addresses were invalid, 102 began taking the survey, and 63 finished. Overall, 1,019 WIC clients completed the survey. Participants who were younger than 18 (n=50), male (n=14), and had an implausible reported height (shorter than 4 feet or taller than 7 feet, n=27)19 were removed from the analytic sample, leaving a total sample of 942 respondents. The Institutional Review Boards of Texas State University and the Texas Department of State Health Services approved this study.

Survey

The survey, developed in English in collaboration with Texas WIC staff, included approximately 130 questions, depending on responses to logic-driven branches. To develop a version of the client survey in Spanish, the English survey was translated to Spanish, back-translated, and discrepancies were reconciled. The survey was implemented using Qualtrics software (2014, Provo, UT). The welcome page briefly described the survey, provided assurances of privacy, and described incentives for survey completion, which included credit for taking a WIC nutrition class and receiving a t-shirt. After giving informed consent, participants were asked if they owned a smartphone. Those who responded with ‘no’ were routed to a thank you page and the survey was discontinued.

The survey was divided into 3 major sections corresponding to health behaviors addressed by the WIC app prototype, including physical activity, healthy eating, and breastfeeding, followed by a set of demographics questions. Each health behavior section asked about current health practices, stage of change, facilitators and barriers to performing the health behavior, belief that app features would help to improve health behaviors, and current use of apps regarding that health behavior. Facilitators and barriers to health behaviors were drawn from focus groups held during the initial phase of the UCD of this prototype app.17 The current study is an analysis of participant response to the physical activity and healthy eating features and does not include breastfeeding.

Current practices regarding physical activity were measured using the Godin leisure-time exercise questionnaire, which creates a physical activity score based on questions about intensity and duration of exercise.20 This score classifies participant activity as insufficiently active, moderately active, or active. For analysis, categories were collapsed into a binary variable (0 = insufficient activity, 1 = moderately active or active). Stage of change for physical activity behaviors was assessed using a 4-question system adapted from Wolf et al. (1 = pre-contemplation, 2 = contemplation, 3 = preparation, 4 = action).21 Survey respondents were asked to indicate on a 5-item Likert scale to what extent they agreed that specific barriers and facilitators to physical activity applied to them personally (1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree); these data were not included in this analysis. Finally, after each of the four app prototype features addressing exercise was displayed (an activity calendar, activity tracker, exercise videos, and resource library), clients were asked whether they agreed that the feature would help them exercise more often (1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree).

Intake of fruits and vegetables was used as an indicator of healthy eating practices and was measured using a brief screener employed by Wolf et al. (fruit and vegetable servings consumed on the previous day, excluding servings of white potatoes, were summed for the final count and included in analysis as a continuous variable).21 Participants were also asked how many family meals they had each week, which was included as a continuous variable. Assessments of stage of change and whether the seven prototype features (a meal calendar, healthy eating tracker, cooking videos, a resource library, shopping list, fruit and vegetable game, and a farmer’s market locator) that addressed healthy eating would help them eat more fruits and vegetables were conducted in the same manner as described for physical activity.

The survey included questions about demographic characteristics. Household size was measured as a continuous variable. Age of participants (0 = 35 years or older, 1 = younger than 35 years), education (0 = high school or less, 1 = post-secondary education), race/ethnicity (White = reference, Black, Hispanic, Other), language used to complete the survey (0 = English, 1 = Spanish), employment status (0 = unemployed, 1 = employed), location of residence (0 = urban, 1 = rural), and body mass index (BMI; < 18.5 = underweight, 18.5 – 24.9 = normal weight, 25 – 29.9 = overweight, > 30 = obese) were coded as categorical variables. Due to few participants having a BMI identifying them as underweight, participants identified as underweight and normal weight were combined and used as the BMI reference group. BMI was not calculated for women who were pregnant (1 = pregnant, 0 = not pregnant). Food security was assessed with the U.S. Household Food Security Survey Module: Six-Item Short Form.22 Food security status was coded as a dichotomous variable (0 = very low or low food security, 1 = marginal or high food security). Current use of physical activity or healthy eating apps were coded as dichotomous variables (0 = never or almost never use, 1 = sometimes or daily use).

Statistical analyses

Latent class analysis, a form of mixture modeling allowing for the classification of unobserved heterogeneity in responses to multiple variables, was used to identify homogenous, mutually exclusive groups of WIC clients based on the extent to which they agreed that prototype features would help them exercise more often or eat more fruits and vegetables.23 To identify the number of classes that best represented the underlying response groups, a series of model fit tests were conducted starting with a single-class model. Model fit indices, including Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), and sample-size adjusted AIC (SSA-AIC), were used to determine whether the inclusion of each additional class provided improved model fit. Model entropy, representing the accuracy of assigning individuals to classes, was also considered. Finally, the Vuong-Lo-Mendell-Ruben (VLMR) likelihood ratio provided a statistical test of whether the estimated model significantly improved model fit compared to a model with one less class. Once the optimal number of latent classes was determined, logistic regression was used to examine differences between classes based on sociodemographic and health-related characteristics. The latent class analysis and logistic regression was conducted with Mplus 7.324 using maximum likelihood estimation with robust standard errors, providing treatment of missing data with maximum likelihood and estimation of standard errors robust to non-normality. All other analyses were conducted using IBM SPSS Statistics for Windows, version 24 (IBM Corp., Armonk, N.Y., USA).

Results

Table 1 presents model fit indices used to identify the optimal number of latent classes based on WIC participants’ responses to the survey questions “this app feature would help me exercise more/eat more fruits and vegetables.” Compared to the 1-class model, the 2-class model had improved AIC, BIC, and SSA-BIC model fit indices; the rate of model fit improvement decreased with the 3-class model. The VLMR test indicated that the 2-class model was a significant improvement on the 1-class model (p < .001), but the 3-class model was not a significantly better fit than the 2-class model (p = 0.8). Thus, based on model fit tests and the necessity of parsimony, the 2-class model was identified as the best description of latent classes.

Table 1: Goodness of fit indices for determining number of latent classes among Texas WIC survey respondents.

Class 1 (strongly agree; 32.9% of the sample) was identified as the group that strongly agreed that app features would help them improve targeted health behaviors; those in class 2 (neutral, agree; 64.8% of the sample) agreed or were neutral regarding whether the app features would help improve health behaviors. Figure 1 shows the distribution of responses to questions asking if using the app features would help respondents improve targeted health behaviors.

jmtm.7.2.2f1.jpg

Figure 1: WIC clients’ agreement that app features would help improve targeted health behaviors. a) Depicts Class 1 (strongly agree) and Class 2 (neutral, agree) feedback regarding physical activity features. b) Depicts Class 1 (strongly agree) and Class 2 (neutral, agree) feedback regarding healthy eating features.

Table 2 includes descriptive statistics for the complete analytic sample as well as by latent class. On average, respondents in the complete sample had approximately 5 household members and consumed 7.4 family meals per week, including 3.5 servings of fruits and vegetables per day. Approximately three out of four respondents were younger than 35 years of age at the time of the survey, which categorizes them as millenials.25 Slightly more than half of participants were Hispanic and the vast majority took the survey in English. Approximately 45% of respondents were employed and the majority were urban-dwellers. Fifty-eight percent were overweight or obese. Sixteen percent of the sample was pregnant. Approximately a third of the sample had completed post-secondary education and a third had marginal or high food security. Two-thirds engaged in at least 150 minutes of moderate or intense physical activity each week. Almost two-thirds of participants used apps for exercise and three-quarters used healthy eating apps. The vast majority recognized the benefits of being physically active and eating fruits and vegetables. Likewise, the majority were in contemplation, preparation, or active stages of change regarding being physically active and eating fruits and vegetables.

Table 2: Description of the overall sample of Texas WIC clients and the 2 latent classes.

The results of the multinomial logistic regression are shown in Table 3; class 2 (neutral, agree) was used as the reference group. Age, pregnancy, current app use, and stage of change regarding exercise were significant predictors of class membership. Specifically, respondents were more likely to be in class 1 (strongly agree) if they were younger than 35 years old (OR = 1.40; CI = 1.05, 1.86), were in contemplation, preparation, or active stages of change regarding exercise (OR = 2.28, CI = 1.26, 4.11), or were currently using apps for exercise (OR = 1.33; CI = 1.02, 1.78) or healthy eating (OR = 1.72, CI = 1.22, 2.42). Respondents were significantly less likely to be in class 1 (strongly agree) if they were pregnant (OR = 0.55, CI = 0.37, 0.82).

Table 3: Predictors of Texas WIC clients’ perceptions of WIC app prototype features.

Discussion

This paper describes an intermediate stage of UCD of an app designed for Texas WIC participants. By investigating variation in responses to the app prototype features, our aim was to identify characteristics of survey respondents associated with the strength of their agreement that the physical activity and healthy eating features would help them improve targeted health behaviors. Importantly, survey respondents’ reactions to the app prototype features were positive. Participants who strongly agreed that the app features would help support behavior changes were more likely to be younger than 35 years of age, in the contemplation, preparation, or action stages of change for the targeted health behaviors, and currently using apps to foster these health behaviors.

The influence of age on class membership is not unexpected. Indeed, millennials, or those born after 1980, are the age group most likely to be continually engaged with smartphones26 and use them for a variety of activities such as searching for jobs and accessing health information.27 One potential avenue for increasing the acceptability of this app among older WIC participants would be to link to or otherwise leverage platforms that have cross-generational appeal.28 In the context of the transtheoretical model, it is also not surprising that participants in contemplation, preparation, or action stages of change were more likely to view the proposed app features as supportive, as they were already interested in improving the targeted health behaviors.29 Similarly, individuals currently using apps are likely more receptive to prototype app features, in general. A surprising finding was that pregnancy was a significant predictor of class membership, with pregnant women being half as likely to be in class 1 (strongly agree). Previous research has reported that, in general, pregnant women are interested in health-related apps with features that are provide information specific to pregnancy, such as pregnancy-related risk factors, gestational weight gain, diet and lifestyle, postpartum depression, social support, and early infant feeding.3032 In light of this, one explanation for the relatively tepid response of pregnant women in this study could be that the features were not specific to dietary and exercise recommendations for pregnancy. Additionally, mobile health interventions targeting pregnant women often suffer from low enrollment and high attrition, suggesting that, in general, pregnancy may be a challenging time to address health behavior change.33 However, firm conclusions about the allure of mobile apps to address health behaviors during pregnancy cannot be drawn due to a dearth of relevant studies.33 Given that health behaviors during pregnancy can have a profound impact on maternal and child health, it is important for an app designed for WIC clients to specifically address the needs of pregnant women to support a healthful pregnancy.

A strength of this study was the use of a large sample of Texas WIC clients who have experience with technology. Sample demographics were somewhat comparable to Texas WIC, with 56% of the sample being Hispanic, compared to 68% in Texas WIC,34 and 58% being overweight or obese, compared to 52% in Texas WIC.35 Limitations include electronic recruitment of individuals who owned a smartphone, resulting in a sample biased towards technology use.

Conclusion

Given the enthusiastic response to the Texas WIC app prototype, it would be tempting to finalize app development by simply incorporating the features described in this study. However, while many factors may impact the ultimate success of public health apps, perhaps a central piece revolves around the needs and preferences of the intended user, framed within the context of his or her specific life’s challenges.13 A user-centered approach to developing technology-based tools, such as the UCD process, is a critical step in ensuring that public health interventions reach their target audiences and elicit desired health outcomes.14 Given the health status of vulnerable populations, such as low income women and children participating in WIC,2 developing and implementing efficacious, evidence-based, population-specific tools and technologies that meet the needs of participants is of paramount importance, and may provide a catalyst to improve health equity by removing barriers to accessing information. In the case of the development of a Texas WIC app, engaging clients who are older, in a pre-contemplation stage of change, or pregnant should occur next, so that their specific needs and preferences can be incorporated into the final version. The iterative process of UCD used in this research may serve as a useful framework for development of public health apps.

Acknowledgements

This research was funded by a grant from the Texas Department of State Health Services (contract number 2014-045584). WIC staff provided input on prototype feature design,17 reviewed survey content, and facilitated survey dissemination via their website.

Disclosures

The Institutional Review Boards of Texas State University (2013E3835) and the Texas Department of State Health Services (14-014) approved this study.

Declaration of Competing Interests

All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: all authors had financial support from the Texas Department of State Health Services for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.

References

1. Legal Momentum. Women and Poverty in America. Available from: https://www.legalmomentum.org/women-and-poverty-america. Accessed December 2016.

2. Adler NE, Stewart J. Health disparities across the lifespan: Meaning, methods, and mechanisms. Ann N Y Acad Sci. 2010;1186:5-23.

3. Eidelman AI, Schanler RJ, Johnston M, et al. Breastfeeding and the use of human milk. Pediatrics. 2012;129(3):e827-e841.

4. Oliveira V, Racine E, Olmsted J, Ghelfi LM. The WIC Program: Background, trends and issues (No. 33847). United States Department of Agriculture, Economic Research Service; 2002.

5. United States Department of Agriculture Food and Nutrition Service. WIC Program: Total Participation. Available from: https://www.fns.usda.gov/sites/default/files/pd/26wifypart.pdf. Accessed July 2017.

6. Special Supplemental Nutrition Program for Women, Infants, and Children. 7 C.F.R. §246.11 1985. Available from: http://www.ecfr.gov/cgi-bin/text-idx?SID=a42889f84f99d56ec18d77c9b463c613&node=7:4.1.1.1.10&rgn=div5#se7.4.246_111. Accessed July 2017.

7. Greenblatt Y, Gomez S, Alleman G, et al. Optimizing nutrition education in WIC: Findings from focus groups with Arizona clients and staff. J Nutr Educ Behav. 2016;48(4):289-294.

8. Cates S, Capogrossi K, Sallack L. WIC Nutrition Education Study: Phase I Report. Alexandria, VA: United States Department of Agriculture, Food and Nutrition Service, Office of Policy Support; 2016. Available from: https://fns-prod.azureedge.net/sites/default/files/ops/WICNutEd-PhaseI.pdf. Accessed June 2017.

9. McHenry G. Evolving technologies change the nature of Internet use. National Telecommunications and Information Administration. Available from: https://www.ntia.doc.gov/blog/2016/evolving-technologies-change-nature-internet-use. Accessed July 2017.

10. Deehy K, Hoger FS, Kallio J, et al. Participant-centered education: Building a new WIC nutrition education model. J Nutr Educ Behav. 2010;42(3S):S39-S46.

11. Schoeppe S, Alley S, Van Lippevelde W, et al. Efficacy of interventions that use apps to improve diet, physical activity and sedentary behaviour: A systematic review. Int J Behav Nutr Phys Act. 2016;13(1):127.

12. Localytics. Helping marketers better understand app user trends. Available from: http://www.localytics.com/resources/app-stickiness-index-q1-2015. Accessed July 2017.

13. Barclay G, Sabina A, Graham G. Population health and technology: Placing people first. Am J Public Health. 2014;104(12):2246-2247.

14. McCurdie T, Taneva S, Casselman M, et al. mHealth Consumer apps: The case for user-centered design. Biomed Instrum Technol Mob Heal. 2012;Suppl:49-56.

15. Vorrink SN, Kort HS, Troosters T, et al. A mobile phone app to stimulate daily physical activity in patients with chronic obstructive pulmonary disease: Development, feasibility, and pilot studies. JMIR mHealth uHealth. 2016;4(1):e11.

16. Texas WIC Nutrition Education Survey – Statewide Report. Texas Department of State Health Services; 2016. Available from: https://www.dshs.texas.gov/wichd/bf/surveysreports.aspx. Accessed July 2017.

17. Biediger-Friedman L, Crixell SH, Silva M, et al. User-centered design of a Texas WIC app: A focus group investigation. Am J Health Behav. 2016;40(4):461-71.

18. Bandura A. Self-efficacy: Toward a unifying theory of behavioral change. Adv Behav Res Ther. 1978;1(4):139-161.

19. Koebnick C, Smith N, Huang K, et al. The prevalence of obesity and obesity-related health conditions in a large multiethnic cohort of young adults in California. Ann Epidemiol. 2012;22(9):609-616.

20. Godin G, Shephard R. Godin leisure-time exercise questionnaire. Med Sci Sport Exerc. 1997;29(6):S36-S38.

21. Wolf RL, Lepore SJ, Vandergrift JL, et al. Knowledge, barriers, and stage of change as correlates of fruit and vegetable consumption among urban and mostly immigrant black men. J Am Diet Assoc. 2008;108(8):1315-1322.

22. United States Department of Agriculture Economic Research Service. Household Food Security Survey Module: Six-Item Short Form. Available from: https://www.ers.usda.gov/media/8282/short2012.pdf. Accessed May 2014.

23. Hagenaars J, McCutcheon A, eds. Applied Latent Class Analysis. Cambridge, UK: Cambridge University Press; 2002.

24. Muthén L, Muthén B. Mplus User’s Guide. Vol 7th ed. Los Angeles, CA: Muthén & Muthén; 2015.

25. Experian. Millennials and Technology: The Natural Order of Things. Available from: http://www.experian.com/blogs/marketing-forward/2014/07/07/millennials-and-technology-the-natural-order-of-things. Accessed July 6, 2017.

26. Nielsen. Millennials are Top Smartphone Users. Available from: http://www.nielsen.com/us/en/insights/news/2016/millennials-are-top-smartphone-users.html. Accessed July 2017.

27. Pew Research Center. U.S. Smartphone Use in 2015. Available from: http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/. Accessed June 2015.

28. Pew Research Center. Social Media Update 2016. Available from: http://www.pewinternet.org/2016/11/11/social-media-update-2016/. Accessed May 2018.

29. Prochaska JO, Velicer WF. The transtheoretical model of health behavior change. Am J Heal Promot. 1997;12(1).

30. Osma J, Barrera AZ, Ramphos E. Are pregnant and postpartum women interested in health-related apps? Implications for the prevention of perinatal depression. Cyberpsychology, Behav Soc Netw. 2016;19(6):412-415.

31. Krishnamurti T, Davis A, Wong-Parodi G, et al. Development and testing of the MyHealthyPregnancy App: A behavioral decision research-based tool for assessing and communicating pregnancy risk. JMIR Mhealth Uhealth. 2017;5(4):e42.

32. Waring ME, Moore Simas TA, Xiao RS, et al. Pregnant women’s interest in a website or mobile application for healthy gestational weight gain. Sex Reprod Healthc. 2014;5(4):182-184.

33. O’Brien CM, Cramp C, Dodd JM. Delivery of dietary and lifestyle interventions in pregnancy: Is it time to promote the use of electronic and mobile health technologies? Semin Reprod Med. 2016;34(2):e22-e27.

34. United States Department of Agriculture Food and Nutrition Services. WIC Racial-Ethnic Group Enrollment Data 2012. Available from: https://www.fns.usda.gov/wic/wic-racial-ethnic-group-enrollment-data-2012. Accessed July 2017.

35. Johnson B, Thorn B, McGill B, et al. WIC Participant and Program Characteristics 2012. Alexandria, VA: United States Department of Agriculture, Food and Nutrition Service; 2013. Available at https://www.fns.usda.gov/wic/women-infants-and-children-wic-participant-and-program-characteristics-2012. Accessed June 2017.

Read More