Pages Menu

Posted on Aug 8, 2017 in Original Article | 0 comments

Assessing Electronic Capture of Patient Reported Outcomes in Prostate Cancer Populations

Assessing Electronic Capture of Patient Reported Outcomes in Prostate Cancer Populations

Leslie D. McIntosh, PhD, MPH1, Anand Mohapatra, MD2*, Marshall Strother, MD3, Goutham Vemana, MD4, Alethea Paradis, MTS5, Kerry Madison, BS5, Anthony Juehne, BA1, Arnold D. Bullock, MD5, Gerald L. Andriole, MD5

1Center for Biomedical Informatics, Washington University in St. Louis, School of Medicine, St. Louis, MO 63110; 2Department of Urology, University of Pittsburgh Medical Center, Pittsburgh, PA 15213; 3Division of Urology, Perelman School of Medicine, Philadelphia, PA 19104; 4Urology Associates of Houston, Houston, TX 77089; 5Division of Urologic Surgery, Washington University in St. Louis, School of Medicine, St. Louis, MO 63110

*Not a receipient of a research scholarship

Corresponding Author:

Note: This research was presented as a poster at the 2013 Annual Meeting of the American Urologic Association under the title “Assessing the Feasibility of Electronic Data Collection for Men with Prostate Cancer”.

Journal MTM 6:2:39–46, 2017


Background: In prostate cancer patients, the commonly-used patient-reported outcome tool to assess quality of life is the EPIC-CP (Expanded Prostate Cancer Index Composite for Clinical Practice).

Aims: To evaluate the feasibility of administering an electronic version of the EPIC-CP and determine preference in providing data by either electronic or pencil-and-paper instrument (PAPI) formats.

Methods: Patients with known prostate cancer who had previously completed a PAPI EPIC-CP were asked to choose between the traditional PAPI version of the EPIC-CP and an electronic version on an iPad. After completing the questionnaire, participants were given a satisfaction questionnaire regarding their experience. At the end of the study, physicians and office staff also completed a satisfaction questionnaire.

Results: Of the 225 men enrolled, 104 chose to complete the PAPI version and 121 chose the electronic version. Patients in the electronic group were younger (63.6 vs. 68.9 years) and rated themselves more computer savvy (7.6/10 vs. 4.8/10). The electronic questionnaire took longer to complete (5.4 vs. 4.0 minutes). Patients found both versions easy to use but reported higher satisfaction with the electronic version (8.5/10 vs. 7.8/10). Number of omitted questions was similar between groups. Physicians and office staff were highly satisfied with the electronic version but cited wireless coverage as a limiting factor.

Conclusion: Patients who chose the electronic EPIC-CP found it easy to use and were highly satisfied with their choice. Randomized-controlled trials are necessary to examine the effects of a computerized questionnaire on patients who would otherwise prefer the PAPI.


With an estimated 220,800 cases diagnosed in 2015 alone, prostate cancer is by far the most commonly diagnosed malignancy in males in the United States.1 Prior to treatment and through continued management of prostate cancer patients, informed discussion of functional outcomes and quality-of-life (QOL) has become standard of care.2,3 Physician engagement with QOL issues is considered so fundamental to effective management that consistent assessment of QOL before and after primary therapy has been proposed as a national quality-of-care metric.4

In this context, a number of patient-reported outcome (PRO) tools have been developed to help clinicians evaluate QOL rapidly and systematically.5 These tools, which traditionally employ the use of a paper-and-pencil instrument (PAPI), are filled out prior to the patient interview. Several studies have shown improved patient-physician communication and patient satisfaction using these PRO tools compared to traditional interviewing techniques alone.69 In prostate cancer patients, the commonly-used PRO tool to assess QOL is the EPIC-CP (Expanded Prostate Cancer Index Composite for Clinical Practice).10

Unfortunately, adoption of these tools in the clinical setting remains poor, especially outside of large academic centers.11 One proposed explanation for poor adoption is the time and effort required for patients and office staff to complete the PAPI questionnaires, calculate results, present these results to the clinician, and transfer all of the obtained data to the patient record.1113 Electronic collection of patient data offers potential solutions to these issues by facilitating rapid analysis, advanced graphical presentation, and effortless incorporation into research databases.6,14 Previous studies in populations of cancer patients have shown that clinical use of such computerized PRO tools is feasible, valid, and generally well-received by patients and physicians.6,12,15 However, concerns regarding the implementation of computerized PRO tools in outpatient urology clinics remain,16 as no study has directly addressed patient preference in the adoption of electronic PRO tools over traditional PAPI assessment methods.

In our study, we evaluated the feasibility of implementing an electronic option for completing EPIC-CP during routine office visits in an outpatient setting with a prostate-cancer-specific population. We specifically assessed patient preference when given a choice between electronic and PAPI input methods and identified features, such as age and self-reported technical proficiency, that made patients more likely to select the electronic PRO tool. We also reported patient and physician satisfaction achieved with both survey delivery methods as well as benefits and challenges encountered during implementation of the electronic method.


We identified patients with prostate cancer who presented for follow-up to three outpatient urology clinics of a single institution. The dates of inclusion were August 2012 to May 2013. Patients were required to be proficient in English and between the ages of 18 and 89. To ensure familiarity with the questionnaire, eligible patients were required to have completed the EPIC-CP during a previous visit. During initial contact in the examination room, patients were approached by a research nurse coordinator for enrollment in the study. All participants were given a choice between the traditional PAPI EPIC-CP (Figure 1) and an electronic version. The patient’s medical record number, date of birth, and first name linked questionnaire responses to demographic information within the medical record. Response times were recorded based on time to complete all survey items and time until data were available in the research database. Surveys were considered incomplete and excluded from further analysis if the patient completed fewer than half of the questions. After completing the EPIC-CP, patients provided anonymous feedback on their experience through a separate satisfaction survey.


Figure 1: Paper and Pencil Instrument Version of the EPIC-CP

Our primary outcomes were patient preferences for data entry methods, time to patient survey completion, number of questions omitted, number of men requiring assistance to complete the questionnaire, self-reported satisfaction, and ease of completion with the method chosen. These were evaluated using the Wilcoxon rank-sum tests and Student’s t-tests using the R software environment.17 Secondary outcomes included qualitative physician and office staff feedback regarding electronic survey implementation, time to entry of results into the research database, questions omitted, and staff error in calculating EPIC-CP section totals. Time to entry of results was compared between groups using a t-test; number of questions omitted was compared using both t-tests and chi-squared tests.


The Expanded Prostate Cancer Index Composite for Clinical Practice (EPIC-CP) is a validated questionnaire completed by the patient providing health-related quality-of-life (HRQOL) measures regarding prostate cancer and its treatment.10 It consists of three multiple choice questions in each of five domains (urinary incontinence, urinary irritative/obstructive, bowel function, sexual function, and vitality/hormonal) plus a summary item regarding urinary bother for a total of 16 items. Patients are asked to provide responses based on their health over the preceding four weeks. All items have four to five possible responses ranging from least to most symptomatic, and all responses except the summary urinary bother item are assigned an integer score ranging from 0 (best) to 4 (worst). Overall scores may be calculated for each domain by adding the scores of the component questions, resulting in possible domain scores ranging from 0 (best) to 12 (worst). Similarly, an overall EPIC-CP score may be calculated by adding scores from the five domains with the response of the summary urinary bother item reported separately.18

Paper and Pencil Instrument (PAPI)

The PAPI EPIC-CP used in this study has been fully described elsewhere.18 Briefly, it consisted of a one-page, single-sided presentation of the EPIC-CP questionnaire. The summary urinary bother item was presented first, followed by the scored items, which were grouped by domain. Each item is visually grouped with a list of possible responses, which patients select by circling or checking corresponding boxes. Corresponding numerical scores were provided next to each response, and blank spaces were left at the end of each domain and at the end of the overall questionnaire with instructions for nursing staff to calculate and fill in the appropriate totals.

PAPI questionnaires were completed by patients in the examining room prior to contact with the physician and response times were measured with a stopwatch. Completed PAPI questionnaires were returned to administrative staff. EPIC-CP section and total scores were calculated and written on the completed PAPI EPIC-CP questionnaires, which were subsequently scanned for the patient record and presented to the physician with the patient’s standard chart. Once scanned, the PAPI EPIC-CP forms were collected by a research coordinator and abstracted into the database in batches.

Electronic Questionnaire

The electronic version was developed at WUSM using LimeSurvey,19 a free and open-source survey application. Electronic questionnaires were administered on a tablet computer (iPad [Apple, Cupertino, CA]) via a secure wireless connection. Questions were presented one at a time in landscape view only, although the size of the text could be modified using pinch-to-zoom gestures. Upon clicking a saved icon on the iPad, an initial screen appeared for the research coordinator, who then completed five items: patient first name (free-text), medical record number (eight-digit numeric entry), patient date of birth (date format), treating physician (radio-button with urology physicians’ names), and survey completion method (single selection item of electronic, paper, or refused). After completing these, the research coordinator would hand the iPad to the participant, who would see a text-display with the survey title and instructions to select ‘Next’ to begin the survey. Upon completion of the survey, EPIC-CP section totals were automatically calculated, which appeared to the participant and research coordinator on the final screen (Figure 1). All information was available to the clinician via a separate web interface, and data from patients who elected to participate in the study were automatically logged in the research database.

Satisfaction Surveys

Upon completion of the EPIC-CP questionnaire, participants were asked to complete an anonymous PAPI questionnaire examining their satisfaction with the questionnaire format, their comfort in completing the questionnaire, and their self-assessed computer literacy. Scores were expressed on a 1-10 scale, with 10 indicating high satisfaction or proficiency. At the conclusion of the study, physicians and office staff were asked to complete a brief questionnaire regarding their experience with the study and both input methods.

Ethics Statement

The protocol for this research project was approved by the Institutional Review Board (201208089) and the Protocol Review Monitoring Committee (12-X135) of our institution. Our study conforms to the provisions of the Declaration of Helsinski (as revised in Tokyo 2008). Informed consent was obtained from all individual participants included in the study.


Two hundred twenty-five men were enrolled in the study. One hundred twenty-one (54%) chose the electronic version of the EPIC-CP questionnaire, and 104 (46%) chose the PAPI version. Three patients who chose the electronic version had incomplete surveys due to technical issues, such as dropped internet access; these were excluded from further analysis. Patients who chose the electronic version were significantly younger than those who chose the PAPI version (mean age 63.6 vs. 68.9, respectively, p<0.0001, Table 1). Those who chose the electronic questionnaire also rated their computer skills more highly (mean 7.6 vs 4.8, respectively, p<0.0001). Additionally, patients who completed the electronic questionnaire were less likely have a relative, friend, or office staff member complete the questionnaire for them (total 1 vs. 8 patients, respectively, p=0.0354). The overall score for the PAPI questionnaire was tallied by hand 34 times, and 13 of these were inaccurate for an error rate of 38.2%. Automatic calculation of section totals for the electronic survey gave accurate results.

Table 1: Number of patients, patient age, completion time, and time until EPIC-CP results in research database separated by questionnaire version.

Participants took longer to complete the electronic survey (mean 5.4 vs. 4.0 minutes, respectively, p=0.0025). However, responses from the electronic questionnaire were uploaded to the research database instantaneously compared to the mean 9.6 days taken to input the PAPI responses (95% CI 8.1–11.1 days). While patients found both formats easy to use (electronic mean 9.1 vs. PAPI mean 8.8, p=0.0612), satisfaction was significantly higher in patients who completed the electronic version (mean 8.5 vs. 7.8, respectively, p=0.0148).

Overall, there was no significant difference between the input methods and the number of questions omitted (electronic mean 0.38 vs. PAPI mean 0.27 questions skipped, p=0.382). For questions 7 to 9, which deal with sexual functioning, there was no significant difference in questions skipped (electronic mean 0.09 vs. PAPI mean 0.20 questions skipped, p=0.14). We did find that patients who took the electronic version may have been more likely to have skipped one item related to sexual functioning while patients completing the PAPI version skipped two or three items, but this did not reach statistical significance (Table 2).

Table 2: Number of questions related to sexual functioning skipped by patients separated by questionnaire version.

There were 7 respondents for the separate satisfaction survey for physicians and office staff to take after the study. All considered both the electronic and PAPI versions easy to use (mean 9.2 vs 8.4, respectively, p=0.62). The most frequent complaint regarding the electronic version was that the office wireless internet coverage was insufficient (n=2).


In this study, we describe the introduction of an electronic version of the EPIC-CP questionnaire for men being followed for prostate cancer in outpatient urology clinics. Approximately half of the men enrolled chose the electronic questionnaire. As expected, those who chose the electronic version were slightly younger and perceived themselves as more skilled with computer technology. These men found the electronic questionnaire easy to use, completed it quickly, and were highly satisfied with their choice. Our study also suggests that patients who express a preference for electronic input methods are capable of using them and ultimately satisfied with their choices regardless of their age or self-reported technical proficiency. Our data demonstrates the feasibility of introducing a successful electronic data collection method for the EPIC-CP questionnaire in urology offices. This is consistent with the literature in other fields of medicine, which shows that electronic questionnaires yield increased user satisfaction and comparable data collection. Examples include questionnaires on pelvic floor dysfunction, general health, pain, and disease-specific quality of life.2026


Figure 2: iPad Summary Screen

Physicians and office staff were also highly satisfied with the electronic surveys. This is especially notable given that accessing the results of the questionnaire on a website separate from the normal medical record required a deviation from the normal workflow. The fact that questionnaire and assessment results could be quickly displayed on a tablet computer and then carried into the examining room was cited as a significant improvement over the presentation of other pieces of clinical data in the EMR, which must be accessed at static workstations in our clinic. The deviation in workflow for office staff may have been offset by the automatic calculation of EPIC-CP section totals in the electronic survey.

The advantage of accurate calculations of section totals merits special attention, as inaccurate section total calculations represents a clear breakdown in communication, leading clinicians to make decisions based on faulty information. However, proper functioning of the electronic questionnaire was contingent on adequate wireless coverage. Because wireless networks at many outpatient clinics are not currently treated as mission-critical resources like the wired connection to the electronic EMR, sites adopting electronic PRO tool options should strongly consider ensuring the availability of a back-up entry method or asynchronous survey collection, so the data will be uploaded as soon as an internet connection is again available.

Those who completed the electronic survey were significantly more satisfied with their experience than those who filled out the PAPI form. Our results may be biased by self-selection by those who are comfortable with computer technology. It also may be reflective of a correlation between affinity for technology and another confounder that could make these patients more satisfied, such as appreciation of quantitative PRO questionnaires themselves. Similar statements can be made about the increased speed and decreased frequency of omitting questions regarding sexual health demonstrated by those choosing the electronic questionnaire. We have begun work on a randomized-controlled trial to examine the effects of a computerized questionnaire on patients who would have otherwise preferred the PAPI.


Both PAPI and electronic EPIC-CP questionnaires were associated with high levels of patient satisfaction regardless of age and self-reported computer skills. The electronic questionnaire may help address potential barriers to adoption of PRO tools. Qualitative assessment by staff highlighted the electronic system’s advantages in data entry and access; however, these were dependent on local wireless infrastructure.

Conflicts of Interest

Financial support was provided by the Midwest Stone Institute. Otherwise, all authors declare no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work.


The authors wish to acknowledge the Midwest Stone Institute for their allotment of project funds via award 80943.


1. Siegel RL, Miller KD, Jemal A. Cancer statistics, 2015. CA: a cancer journal for clinicians. 2015;65(1):5–29.

2. Mohler J, Bahnson RR, Boston B, Busby JE, D’Amico A, Eastham JA, et al. NCCN clinical practice guidelines in oncology: prostate cancer. Journal of the National Comprehensive Cancer Network : JNCCN. 2010;8(2):162–200.

3. Thompson I, Thrasher JB, Aus G, Burnett AL, Canby-Hagino ED, Cookson MS, et al. Guideline for the management of clinically localized prostate cancer: 2007 update. The Journal of urology. 2007;177(6):2106–31.

4. Litwin MS. Prostate cancer patient outcomes and choice of providers : development of an infrastructure for quality assessment. Santa, Monica, Calif.: Rand; 2000. ix, 248 p.

5. Rnic K, Linden W, Tudor I, Pullmer R, Vodermaier A. Measuring symptoms in localized prostate cancer: a systematic review of assessment instruments. Prostate cancer and prostatic diseases. 2013;16(2):111–22.

6. Berry DL, Blumenstein BA, Halpenny B, Wolpin S, Fann JR, Austin-Seymour M, et al. Enhancing patient-provider communication with the electronic self-report assessment for cancer: a randomized trial. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. 2011;29(8):1029–35.

7. Sonn GA, Sadetsky N, Presti JC, Litwin MS. Differing perceptions of quality of life in patients with prostate cancer and their doctors. The Journal of urology. 2013;189(1 Suppl):S59–65; discussion S.

8. Detmar SB, Muller MJ, Schornagel JH, Wever LD, Aaronson NK. Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial. JAMA : the journal of the American Medical Association. 2002;288(23):3027–34.

9. Lipscomb J, Gotay CC, Snyder CF. Patient-reported outcomes in cancer: a review of recent research and policy initiatives. CA Cancer J Clin. 2007;57(5):278–300.

10. Chang P, Szymanski KM, Dunn RL, Chipman JJ. Expanded prostate cancer index composite for clinical practice: development and validation of a practical health related quality of life instrument for use in the routine …. The Journal of. …. 2011.

11. Kim SP, Karnes RJ, Nguyen PL, Ziegenfuss JY, Han LC, Thompson RH, et al. Clinical implementation of quality of life instruments and prediction tools for localized prostate cancer: results from a national survey of radiation oncologists and urologists. The Journal of urology. 2013;189(6):2092–8.

12. Berry DL, Trigg LJ, Lober WB, Karras BT, Galligan ML, Austin-Seymour M, et al. Computerized symptom and quality-of-life assessment for patients with cancer part I: development and pilot testing. Oncology nursing forum. 2004;31(5):E75–83.

13. Hacker ED. Technology and quality of life outcomes. Seminars in oncology nursing. 2010;26(1):47–58.

14. Rose M, Bezjak A. Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples. Quality of life research: an international journal of quality of life aspects of treatment, care and rehabilitation. 2009;18(1):125–36.

15. Seow H, King S, Green E, Pereira J, Sawka C. Perspectives of patients on the utility of electronic patient-reported outcomes on cancer care. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. 2011;29(31):4213–4.

16. Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why? Quality of Life Research. 2009.

17. Team RC. R: A Language and environment for statistical computing. Vienna, Austria: the R Foundation for Statistical Computing; 2012.

18. Chang P, Szymanski KM, Dunn RL, Chipman JJ, Litwin MS, Nguyen PL, et al. Expanded Prostate Cancer Index Composite for Clinical Practice: Development and Validation of a Practical Health Related Quality of Life Instrument for Use in the Routine Clinical Care of Patients With Prostate Cancer. J Urology. 2011;186(3):865–72.

19. Goyer LPTCSMMTPDCN. / LimeSurvey: An Open Source survey tool: Alexa The Web Information Company; 2012. Available from:

20. Bushnell DM, Reilly MC, Galani C, Martin ML, Ricci J-FF, Patrick DL, et al. Validation of electronic data capture of the Irritable Bowel Syndrome–Quality of Life Measure, the Work Productivity and Activity Impairment Questionnaire for Irritable Bowel Syndrome and the EuroQol. Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research. 2006;9(2):98–105.

21. Cook AJ, Roberts DA, Henderson MD, Winkle VLC. Electronic pain questionnaires: a randomized, crossover comparison with paper questionnaires for chronic pain assessment. Pain. 2004.

22. Duracinsky M, Lalanne C, Goujard C. Electronic versus paper-based assessment of health-related quality of life specific to HIV disease: reliability study of the PROQOL-HIV questionnaire. Journal of medical. …. 2014.

23. Handa VL, Barber MD, Young SB, Aronson MP. Paper versus web-based administration of the pelvic floor distress inventory 20 and pelvic floor impact questionnaire 7. International. …. 2008.

24. Juniper EF, Riis B, Juniper BA. Development and validation of an electronic version of the Rhinoconjunctivitis Quality of Life Questionnaire. Allergy. 2007.

25. Parnell BA, Dunivan GC, Connolly AM. Validation of web-based administration of the pelvic organ prolapse/urinary incontinence sexual function questionnaire (PISQ-12). International. …. 2011.

26. Ryan JM, Corry JR, Attewell R, Smithson MJ. A comparison of an electronic version of the SF-36 General Health Questionnaire to the standard paper version. Quality of life research. 2002.