Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study

Kocaballi, Ahmet Baki; Sezgin, Emre; Clark, Leigh; Carroll, John M.; Huang, Yungui; Huh-Yoo, Jina; Kim, Junhan; Kocielnik, Rafal; Lee, Yi-Chieh; Mamykina, Lena; Mitchell, Elliot G.; Moore, Robert J.; Murali, Prasanth; Mynatt, Elizabeth D.; Park, Sun Young; Pasta, Alessandro; Richards, Deborah; Silva, Lucas M.; Smriti, Diva; Spillane, Brendan; Zhang, Zhan; Zubatiy, Tamara

doi:10.2196/38525

Published November 15, 2022 | Version Published

Journal Article Open

Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study

1. University of Technology Sydney
2. Macquarie University
3. Nationwide Children's Hospital
4. Pennsylvania State University
5. Drexel University
6. University of Michigan–Ann Arbor
7. California Institute of Technology
8. National University of Singapore
9. Columbia University
10. IBM Research - Almaden
11. Northeastern University
12. Technical University of Denmark
13. University of California, Irvine
14. University College Dublin
15. Pace University
16. Georgia Institute of Technology

Background: Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged. Objective: This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges. Methods: A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies. Results: The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence. Conclusions: CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.

Additional Information

© Ahmet Baki Kocaballi, Emre Sezgin, Leigh Clark, John M Carroll, Yungui Huang, Jina Huh-Yoo, Junhan Kim, Rafal Kocielnik, Yi-Chieh Lee, Lena Mamykina, Elliot G Mitchell, Robert J Moore, Prasanth Murali, Elizabeth D Mynatt, Sun Young Park, Alessandro Pasta, Deborah Richards, Lucas M Silva, Diva Smriti, Brendan Spillane, Zhan Zhang, Tamara Zubatiy. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 15.11.2022. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included. Authors' Contributions: ABK contributed to the study design. ABK, ES, LC, JMC, JH-Y, YH, JK, RK, Y-CL, LM, EGM, RJM, PM, EDM, SYP, AP, DR, LMS, DS, BS, ZZ, and TZ contributed to data reporting. ABK, ES, and LC performed thematic analysis. ABK, ES, and LC proposed the first draft. ABK, ES, LC, JMC, JH-Y, YH, JK, RK, Y-CL, LM, EGM, RJM, PM, EDM, SYP, AP, DR, LMS, DS, BS, ZZ, and TZ performed revisions and finalized subsequent drafts. Conflicts of Interest: None declared.

Attached Files

Published - Design_and_Evaluation_Challenges_of_Conversational_Agents_in_Health_Care_and_Well-being__Selective_Review_Study_-_PMC.pdf

Files