Journal Article2022

Design Validation of a Relational Agent by COVID-19 Patients

Ashraful Islam, Beenish Moalla Chaudhry

JMIR Human Factors

JMIR Publications, Vol. 10, Issue Forthcoming, pp. e42740, ISBN: 2292-9495

DOI: 10.2196/42740URL

Abstract

Background Relational agents (RAs) have shown effectiveness in various health interventions with and without doctors and hospital facilities. In situations such as a pandemic like the COVID-19 pandemic when health care professionals (HCPs) and facilities are unable to cope with increased demands, RAs may play a major role in ameliorating the situation. However, they have not been well explored in this domain. Objective This study aimed to design a prototypical RA in collaboration with COVID-19 patients and HCPs and test it with the potential users, for its ability to deliver services during a pandemic. Methods The RA was designed and developed in collaboration with people with COVID-19 (n=21) and 2 groups of HCPs (n=19 and n=16, respectively) to aid COVID-19 patients at various stages by performing 4 main tasks: testing guidance, support during self-isolation, handling emergency situations, and promoting postrecovery mental well-being. A design validation survey was conducted with 98 individuals to evaluate the usability of the prototype using the System Usability Scale (SUS), and the participants provided feedback on the design. In addition, the RA’s usefulness and acceptability were rated by the participants using Likert scales. Results In the design validation survey, the prototypical RA received an average SUS score of 58.82. Moreover, 90% (88/98) of participants perceived it to be helpful, and 69% (68/98) of participants accepted it as a viable alternative to HCPs. The prototypical RA received favorable feedback from the participants, and they were inclined to accept it as an alternative to HCPs in non-life-threatening scenarios despite the usability rating falling below the acceptable threshold. Conclusions Based on participants’ feedback, we recommend further development of the RA with improved automation and emotional support, ability to provide information, tracking, and specific recommendations.

CCDS Authors

Ashraful Islam

Ashraful Islam, PhD

Center DirectorDirector, Human-Computer Interaction Wing

References

  1. 1.Virginia Braun, Victoria Clarke. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101[10.1191/1478088706qp063oa]
  2. 2.Aaron Bangor, Philip Kortum, James Miller. (2008). An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24(6), 574–594[10.1080/10447310802205776]
  3. 3.Adam O. Horvath, Leslie S. Greenberg. (1989). Development and validation of the Working Alliance Inventory.. Journal of Counseling Psychology, 36(2), 223–233[10.1037/0022-0167.36.2.223]
  4. 4.Stewart W Mercer. (2004). The consultation and relational empathy (CARE) measure: development and preliminary validation and reliability of an empathy-based consultation process measure. Family Practice, 21(6), 699–705[10.1093/fampra/cmh621]
  5. 5.Timothy Bickmore, Justine Cassell. (2001). Relational agents. , 396–403[10.1145/365024.365304]
  6. 6.Cynthia L. Bennett, Daniela K. Rosner. (2019). The Promise of Empathy. , 1–13[10.1145/3290605.3300528]
  7. 7.Timothy Bickmore, Lisa B. Caruso, Kerri M. Clough‐Gorr, Timothy Heeren. (2005). ‘It's just like you talk to a friend’ relational agents for older adults. Interacting with Computers, 17(6), 711–735[10.1016/j.intcom.2005.09.002]
  8. 8.Susanne Bødker, Morten Kyng. (2018). Participatory Design that Matters—Facing the Big Issues. ACM Transactions on Computer-Human Interaction, 25(1), 1–31[10.1145/3152421]
  9. 9.Adam S. Miner, Liliana Laranjo, A. Baki Kocaballı. (2020). Chatbots in the fight against the COVID-19 pandemic. npj Digital Medicine, 3(1), 65[10.1038/s41746-020-0280-0]
  10. 10.Silke ter Stal, Lean L Kramer, Monique Tabak, Harm op den Akker, Hermie Hermens. (2020). Design Features of Embodied Conversational Agents in eHealth: a Literature Review. International Journal of Human-Computer Studies, 138, 102409[10.1016/j.ijhcs.2020.102409]
  11. 11.Gopi Battineni, Nalini Chintalapudi, Francesco Amenta. (2020). AI Chatbot Design during an Epidemic like the Novel Coronavirus. Healthcare, 8(2), 154[10.3390/healthcare8020154]
  12. 12.Lean L Kramer, Silke ter Stal, Bob C Mulder, Emely de Vet, Lex van Velsen. (2019). Developing Embodied Conversational Agents for Coaching People in a Healthy Lifestyle: Scoping Review. Journal of Medical Internet Research, 22(2), e14058[10.2196/14058]
  13. 13.Katherine Easton, Stephen Potter, Rémi Bec, Matthew Russell Bennion, Heidi Christensen, Cheryl Grindell, Bahman Mirheidari, Scott Weich, Luc de Witte, Daniel Wolstenholme, Mark Hawley. (2019). A Virtual Agent to Support Individuals Living With Physical and Mental Comorbidities: Co-Design and Acceptability Testing. Journal of Medical Internet Research, 21(5), e12996[10.2196/12996]
  14. 14.Myrthe L. Tielman, Mark A. Neerincx, Rafael Bidarra, Ben Kybartas, Willem‐Paul Brinkman. (2017). A Therapy System for Post-Traumatic Stress Disorder Using a Virtual Agent and Virtual Storytelling to Reconstruct Traumatic Memories. Journal of Medical Systems, 41(8), 125[10.1007/s10916-017-0771-y]
  15. 15.Kate Loveys, Mark Sagar, Isabella Pickering, Elizabeth Broadbent. (2021). A Digital Human for Delivering a Remote Loneliness and Stress Intervention to At-Risk Younger and Older Adults During the COVID-19 Pandemic: Randomized Pilot Trial. JMIR Mental Health, 8(11), e31586[10.2196/31586]
  16. 16.Onil Bhattacharyya, Kathryn Mossman, Lovisa Gustafsson, Eric C. Schneider. (2018). Using Human-Centered Design to Build a Digital Health Advisor for Patients With Complex Needs: Persona and Prototype Development. Journal of Medical Internet Research, 21(5), e10318[10.2196/10318]
  17. 17.Marie A. Sillice, Patricia J. Morokoff, Ginette G. Ferszt, Timothy Bickmore, Beth C. Bock, Ryan Lantini, Wayne F. Velicer. (2018). Using Relational Agents to Promote Exercise and Sun Protection: Assessment of Participants’ Experiences With Two Interventions. Journal of Medical Internet Research, 20(2), e48[10.2196/jmir.7640]
  18. 18.Nick Bott, Sharon Stahl Wexler, Lin J. Drury, Chava Pollak, Victor Wang, Kathleen Scher, Sharon Narducci. (2019). A Protocol-Driven, Bedside Digital Conversational Agent to Support Nurse Teams and Mitigate Risks of Hospitalization in Older Adults: Case Control Pre-Post Study. Journal of Medical Internet Research, 21(10), e13440[10.2196/13440]
  19. 19.Simon Galmiche, Eve Rahbé, Arnaud Fontanet, Aurélien Dinh, François Bénézit, François-Xavier Lescure, Fabrice Denis. (2020). Implementation of a Self-Triage Web Application for Suspected COVID-19 and Its Impact on Emergency Call Centers: Observational Study. Journal of Medical Internet Research, 22(11), e22924[10.2196/22924]
  20. 20.Davy Paap, Ernst Schrier, Pieter U. Dijkstra. (2018). Development and validation of the Working Alliance Inventory Dutch version for use in rehabilitation setting. Physiotherapy Theory and Practice, 35(12), 1292–1303[10.1080/09593985.2018.1471112]
  21. 21.Erinç Merdivan, Deepika Singh, Sten Hanke, Andreas Holzinger. (2019). Dialogue Systems for Intelligent Human Computer Interactions. Electronic Notes in Theoretical Computer Science, 343, 57–71[10.1016/j.entcs.2019.04.010]
  22. 22.Mollie M. McKillop, Brett R. South, Anita M. Preininger, Mitch Mason, Gretchen Purcell Jackson. (2020). Leveraging conversational technology to answer common COVID-19 questions. Journal of the American Medical Informatics Association, 28(4), 850–855[10.1093/jamia/ocaa316]
  23. 23.Junhan Kim, Jana Muhic, Lionel Robert, Sun Young Park. (2022). Designing Chatbots with Black Americans with Chronic Conditions: Overcoming Challenges against COVID-19. CHI Conference on Human Factors in Computing Systems, 1–17[10.1145/3491102.3502116]
  24. 24.Myrthe L. Tielman, Mark A. Neerincx, Willem‐Paul Brinkman. (2019). Design and Evaluation of Personalized Motivational Messages by a Virtual Agent that Assists in Post-Traumatic Stress Disorder Therapy. Journal of Medical Internet Research, 21(3), e9240[10.2196/jmir.9240]
  25. 25.Sophia Siedlikowski, Louis-Philippe Noël, Stephanie Anne Moynihan, Marc R. Robin. (2021). Chloe for COVID-19: Evolution of an Intelligent Conversational Agent to Address Infodemic Management Needs During the COVID-19 Pandemic. Journal of Medical Internet Research, 23(9), e27283[10.2196/27283]
  26. 26.Wai Lok Woo, Bin Gao, Raid Rafi Omar Al-Nima, Bingo Wing‐Kuen Ling. (2020). Development of Conversational Artificial Intelligence for Pandemic Healthcare Query Support. International Journal of Automation Artificial Intelligence and Machine Learning, 54–79[10.61797/ijaaiml.v1i1.35]
  27. 27.Geicianfran da Silva Lima Roque, Andréia Roque de Souza Cavalcanti, José William Araújo do Nascimento, Rafael Roque de Souza, Sergio Queiroz. (2021). BotCovid: Development and Evaluation of a Chatbot to Combat Misinformation about COVID-19 in Brazil. 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2506–2511[10.1109/smc52423.2021.9658693]
  28. 28.Ashraful Islam, Beenish Moalla Chaudhry. (2021). Identifying User Personas for Engagement with a COVID-19 Health Service Delivery Relational Agent. , 364–366[10.1145/3472307.3484658]
  29. 29.Beenish Moalla Chaudhry, Ashraful Islam. (2022). A Mobile Application-Based Relational Agent as a Health Professional for COVID-19 Patients: Design, Approach, and Implications. International Journal of Environmental Research and Public Health, 19(21), 13794[10.3390/ijerph192113794]
  30. 30.Simon George Taukeni. (2019). Psychology of Health - Biopsychosocial Approach. IntechOpen eBooks[10.5772/intechopen.79036]
  31. 31.Nikoletta Ventoura, Kosmas Palios, Yannis Vasilakis, Georgios Paraskevopoulos, Nassos Katsamanis, Vassilis Katsouros. (2021). Theano: A Greek-speaking conversational agent for COVID-19. , 36–46[10.18653/v1/2021.nlp4posimpact-1.5]
  32. 32.Etsuko Ishii, Genta Indra Winata, Samuel Cahyawijaya, Divesh Lala, Tatsuya Kawahara, Pascale Fung. (2021). ERICA: An Empathetic Android Companion for Covid-19 Quarantine. , 257–260[10.18653/v1/2021.sigdial-1.27]
← All publications