Journal Article2023Cited by 9

Design Validation of a Relational Agent by COVID-19 Patients: Mixed Methods Study

Ashraful Islam, Beenish Moalla Chaudhry

JMIR Human Factors

JMIR Publications Toronto, Canada, Vol. 10, pp. e42740, ISBN: 2292-9495

  • Good Health and Well-being — Relevance: 81%

Abstract

Background Relational agents (RAs) have shown effectiveness in various health interventions with and without doctors and hospital facilities. In situations such as a pandemic like the COVID-19 pandemic when health care professionals (HCPs) and facilities are unable to cope with increased demands, RAs may play a major role in ameliorating the situation. However, they have not been well explored in this domain. Objective This study aimed to design a prototypical RA in collaboration with COVID-19 patients and HCPs and test it with the potential users, for its ability to deliver services during a pandemic. Methods The RA was designed and developed in collaboration with people with COVID-19 (n=21) and 2 groups of HCPs (n=19 and n=16, respectively) to aid COVID-19 patients at various stages by performing 4 main tasks: testing guidance, support during self-isolation, handling emergency situations, and promoting postrecovery mental well-being. A design validation survey was conducted with 98 individuals to evaluate the usability of the prototype using the System Usability Scale (SUS), and the participants provided feedback on the design. In addition, the RA’s usefulness and acceptability were rated by the participants using Likert scales. Results In the design validation survey, the prototypical RA received an average SUS score of 58.82. Moreover, 90% (88/98) of participants perceived it to be helpful, and 69% (68/98) of participants accepted it as a viable alternative to HCPs. The prototypical RA received favorable feedback from the participants, and they were inclined to accept it as an alternative to HCPs in non-life-threatening scenarios despite the usability rating falling below the acceptable threshold. Conclusions Based on participants’ feedback, we recommend further development of the RA with improved automation and emotional support, ability to provide information, tracking, and specific recommendations.

Keywords

  • relational agents
  • COVID-19 pandemic
  • System usability scale
  • chatbots
  • healthcare workers
  • design validation
  • usability evaluation
  • patient acceptance

CCDS Authors

Ashraful Islam, PhD
Ashraful Islam

Ashraful Islam, PhD

  • Center Director, CCDS
  • Director, HCI Wing

Assistant Professor

Department of Computer Science and Engineering

Independent University, Bangladesh (IUB)

Published at

CHI, UIST, UbiComp, IJCNN, ICIP, HCII, ICDMW, BLP, IEEE Internet of Things Journal, JMIR, JMIR Human Factors, JMIR Formative Research, IEEE Access, Smart Agricultural Technology, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Human-Computer Interaction, AI for Social Good, AI for Public Health, AI for Impact

Beenish Moalla Chaudhry, PhD
Beenish Moalla Chaudhry, PhD

Beenish Moalla Chaudhry, PhD

  • Collaborator, HCI Wing

Associate Professor

School of Computing and Informatics

University of Louisiana at Lafayette

Published at

UIST, HCII, JMIR Human Factors, JMIR Formative Research

References

  1. 1.Virginia Braun, Victoria Clarke. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101[10.1191/1478088706qp063oa]
  2. 2.Aaron Bangor, Philip Kortum, James Miller. (2008). An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24(6), 574–594[10.1080/10447310802205776]
  3. 3.Adam O. Horvath, Leslie S. Greenberg. (1989). Development and validation of the Working Alliance Inventory.. Journal of Counseling Psychology, 36(2), 223–233[10.1037/0022-0167.36.2.223]
  4. 4.Stewart W Mercer. (2004). The consultation and relational empathy (CARE) measure: development and preliminary validation and reliability of an empathy-based consultation process measure. Family Practice, 21(6), 699–705[10.1093/fampra/cmh621]
  5. 5.Timothy Bickmore, Justine Cassell. (2001). Relational agents. , 396–403[10.1145/365024.365304]
  6. 6.Cynthia L. Bennett, Daniela K. Rosner. (2019). The Promise of Empathy. , 1–13[10.1145/3290605.3300528]
  7. 7.Timothy Bickmore, Lisa B. Caruso, Kerri M. Clough‐Gorr, Timothy Heeren. (2005). ‘It's just like you talk to a friend’ relational agents for older adults. Interacting with Computers, 17(6), 711–735[10.1016/j.intcom.2005.09.002]
  8. 8.Susanne Bødker, Morten Kyng. (2018). Participatory Design that Matters—Facing the Big Issues. ACM Transactions on Computer-Human Interaction, 25(1), 1–31[10.1145/3152421]
  9. 9.Adam S. Miner, Liliana Laranjo, A. Baki Kocaballı. (2020). Chatbots in the fight against the COVID-19 pandemic. npj Digital Medicine, 3(1), 65[10.1038/s41746-020-0280-0]
  10. 10.Silke ter Stal, Lean L Kramer, Monique Tabak, Harm op den Akker, Hermie Hermens. (2020). Design Features of Embodied Conversational Agents in eHealth: a Literature Review. International Journal of Human-Computer Studies, 138, 102409[10.1016/j.ijhcs.2020.102409]
  11. 11.Gopi Battineni, Nalini Chintalapudi, Francesco Amenta. (2020). AI Chatbot Design during an Epidemic like the Novel Coronavirus. Healthcare, 8(2), 154[10.3390/healthcare8020154]
  12. 12.Lean L Kramer, Silke ter Stal, Bob C Mulder, Emely de Vet, Lex van Velsen. (2019). Developing Embodied Conversational Agents for Coaching People in a Healthy Lifestyle: Scoping Review. Journal of Medical Internet Research, 22(2), e14058[10.2196/14058]
  13. 13.Katherine Easton, Stephen Potter, Rémi Bec, Matthew Russell Bennion, Heidi Christensen, Cheryl Grindell, Bahman Mirheidari, Scott Weich, Luc de Witte, Daniel Wolstenholme, Mark Hawley. (2019). A Virtual Agent to Support Individuals Living With Physical and Mental Comorbidities: Co-Design and Acceptability Testing. Journal of Medical Internet Research, 21(5), e12996[10.2196/12996]
  14. 14.Myrthe L. Tielman, Mark A. Neerincx, Rafael Bidarra, Ben Kybartas, Willem‐Paul Brinkman. (2017). A Therapy System for Post-Traumatic Stress Disorder Using a Virtual Agent and Virtual Storytelling to Reconstruct Traumatic Memories. Journal of Medical Systems, 41(8), 125[10.1007/s10916-017-0771-y]
  15. 15.Kate Loveys, Mark Sagar, Isabella Pickering, Elizabeth Broadbent. (2021). A Digital Human for Delivering a Remote Loneliness and Stress Intervention to At-Risk Younger and Older Adults During the COVID-19 Pandemic: Randomized Pilot Trial. JMIR Mental Health, 8(11), e31586[10.2196/31586]
  16. 16.Onil Bhattacharyya, Kathryn Mossman, Lovisa Gustafsson, Eric C. Schneider. (2018). Using Human-Centered Design to Build a Digital Health Advisor for Patients With Complex Needs: Persona and Prototype Development. Journal of Medical Internet Research, 21(5), e10318[10.2196/10318]
  17. 17.Marie A. Sillice, Patricia J. Morokoff, Ginette G. Ferszt, Timothy Bickmore, Beth C. Bock, Ryan Lantini, Wayne F. Velicer. (2018). Using Relational Agents to Promote Exercise and Sun Protection: Assessment of Participants’ Experiences With Two Interventions. Journal of Medical Internet Research, 20(2), e48[10.2196/jmir.7640]
  18. 18.Nick Bott, Sharon Stahl Wexler, Lin J. Drury, Chava Pollak, Victor Wang, Kathleen Scher, Sharon Narducci. (2019). A Protocol-Driven, Bedside Digital Conversational Agent to Support Nurse Teams and Mitigate Risks of Hospitalization in Older Adults: Case Control Pre-Post Study. Journal of Medical Internet Research, 21(10), e13440[10.2196/13440]
  19. 19.Simon Galmiche, Eve Rahbé, Arnaud Fontanet, Aurélien Dinh, François Bénézit, François-Xavier Lescure, Fabrice Denis. (2020). Implementation of a Self-Triage Web Application for Suspected COVID-19 and Its Impact on Emergency Call Centers: Observational Study. Journal of Medical Internet Research, 22(11), e22924[10.2196/22924]
  20. 20.Davy Paap, Ernst Schrier, Pieter U. Dijkstra. (2018). Development and validation of the Working Alliance Inventory Dutch version for use in rehabilitation setting. Physiotherapy Theory and Practice, 35(12), 1292–1303[10.1080/09593985.2018.1471112]
  21. 21.Erinç Merdivan, Deepika Singh, Sten Hanke, Andreas Holzinger. (2019). Dialogue Systems for Intelligent Human Computer Interactions. Electronic Notes in Theoretical Computer Science, 343, 57–71[10.1016/j.entcs.2019.04.010]
  22. 22.Mollie M. McKillop, Brett R. South, Anita M. Preininger, Mitch Mason, Gretchen Purcell Jackson. (2020). Leveraging conversational technology to answer common COVID-19 questions. Journal of the American Medical Informatics Association, 28(4), 850–855[10.1093/jamia/ocaa316]
  23. 23.Junhan Kim, Jana Muhic, Lionel Robert, Sun Young Park. (2022). Designing Chatbots with Black Americans with Chronic Conditions: Overcoming Challenges against COVID-19. CHI Conference on Human Factors in Computing Systems, 1–17[10.1145/3491102.3502116]
  24. 24.Myrthe L. Tielman, Mark A. Neerincx, Willem‐Paul Brinkman. (2019). Design and Evaluation of Personalized Motivational Messages by a Virtual Agent that Assists in Post-Traumatic Stress Disorder Therapy. Journal of Medical Internet Research, 21(3), e9240[10.2196/jmir.9240]
  25. 25.Sophia Siedlikowski, Louis-Philippe Noël, Stephanie Anne Moynihan, Marc R. Robin. (2021). Chloe for COVID-19: Evolution of an Intelligent Conversational Agent to Address Infodemic Management Needs During the COVID-19 Pandemic. Journal of Medical Internet Research, 23(9), e27283[10.2196/27283]
  26. 26.Wai Lok Woo, Bin Gao, Raid Rafi Omar Al-Nima, Bingo Wing‐Kuen Ling. (2020). Development of Conversational Artificial Intelligence for Pandemic Healthcare Query Support. International Journal of Automation Artificial Intelligence and Machine Learning, 54–79[10.61797/ijaaiml.v1i1.35]
  27. 27.Geicianfran da Silva Lima Roque, Andréia Roque de Souza Cavalcanti, José William Araújo do Nascimento, Rafael Roque de Souza, Sergio Queiroz. (2021). BotCovid: Development and Evaluation of a Chatbot to Combat Misinformation about COVID-19 in Brazil. 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2506–2511[10.1109/smc52423.2021.9658693]
  28. 28.Ashraful Islam, Beenish Moalla Chaudhry. (2021). Identifying User Personas for Engagement with a COVID-19 Health Service Delivery Relational Agent. , 364–366[10.1145/3472307.3484658]
  29. 29.Beenish Moalla Chaudhry, Ashraful Islam. (2022). A Mobile Application-Based Relational Agent as a Health Professional for COVID-19 Patients: Design, Approach, and Implications. International Journal of Environmental Research and Public Health, 19(21), 13794[10.3390/ijerph192113794]
  30. 30.Simon George Taukeni. (2019). Psychology of Health - Biopsychosocial Approach. IntechOpen eBooks[10.5772/intechopen.79036]
  31. 31.Nikoletta Ventoura, Kosmas Palios, Yannis Vasilakis, Georgios Paraskevopoulos, Nassos Katsamanis, Vassilis Katsouros. (2021). Theano: A Greek-speaking conversational agent for COVID-19. , 36–46[10.18653/v1/2021.nlp4posimpact-1.5]
  32. 32.Etsuko Ishii, Genta Indra Winata, Samuel Cahyawijaya, Divesh Lala, Tatsuya Kawahara, Pascale Fung. (2021). ERICA: An Empathetic Android Companion for Covid-19 Quarantine. , 257–260[10.18653/v1/2021.sigdial-1.27]

Cited By

  1. 1.Yining Hua, Steve Siddals, Zilin Ma, Isaac Galatzer‐Levy, Winna Xia, Christine Hau, Hongbin Na, Matthew Flathers, Jake Linardon, Cyrus Ayubcha, John Torous. (2025). Charting the evolution of artificial intelligence mental health chatbots from rule‐based systems to large language models: a systematic review. World Psychiatry, 24(3), 383–394[10.1002/wps.21352]
  2. 2.Mantapond Ittarat, Wisit Cheungpasitporn, Sunee Chansangpetch. (2023). Personalized Care in Eye Health: Exploring Opportunities, Challenges, and the Road Ahead for Chatbots. Journal of Personalized Medicine, 13(12), 1679[10.3390/jpm13121679]
  3. 3.Virginia Arnold, Tina D Purnat, Robert Marten, Andrew Pattison, Hebe Gouda. (2024). Chatbots and COVID-19: Taking Stock of the Lessons Learned. Journal of Medical Internet Research, 26, e54840[10.2196/54840]
  4. 4.Ashraful Islam, Beenish Moalla Chaudhry, Aminul Islam. (2023). Designing Healthcare Relational Agents: A Conceptual Framework with User-Centered Design Guidelines. , 1–4[10.1109/iscc58397.2023.10218039]
  5. 5.Yingjia Qian, Wenjuan Tang, Tingting Xu, Daqiao Zhu, Wenzhe Hua. (2025). Emotional support powered by artificial intelligence in healthcare settings: A scoping review of technologies, contents, and outcomes. International Journal of Nursing Studies, 172, 105243[10.1016/j.ijnurstu.2025.105243]
  6. 6.Yanrong Yang, Jorge Tavares, Tiago Oliveira. (2026). Exploring user engagement with AI-enabled chatbot for well-being: A cross-national survey in the United States and China. International Journal of Information Management Data Insights, 6(1), 100413[10.1016/j.jjimei.2026.100413]
  7. 7.Mohammad Masudur Rahman, Beenish Moalla Chaudhry. (2026). Ethical Sensemaking in AI Mental Health Chatbots: An Analysis of User Reviews. , 2696–2708[10.1145/3800645.3813050]
  8. 8.João Pavão, Rute Bastardo, Anabela Gonçalves Silva, Nelson Pacheco Rocha. (2026). Usability and User Experience Assessment of Health Care Conversational Agents Using Validated Subjective Instruments: Systematic Review and Comparative Analysis. JMIR Human Factors, 13, e102911[10.2196/102911]
  9. 9.Rene Rathmann, Aylin Bozkurt, Anna Kaindl, Alexander Piazza, Ritam Garg. (2026). Hallo, wie kann ich helfen? Anwendungsfelder und Zukunftsperspektiven von Conversational Agents im Krankenhauskontext. Edition HMD, 423–453[10.1007/978-3-658-51795-3_19]