Conference Paper2025

A Benchmark and Optimization Framework for ML and LLM-Based Health Triage

Md Jobayer Rahman, Shamim Ahamed, Fahim Faisal Deepto, Marzia Zaman, Ashraful Islam, Farhana Sarker, Khondaker A Mamun

2025 IEEE International Conference on Data Mining Workshops (ICDMW)

IEEE, pp. 568-575

  • Good Health and Well-being — Relevance: 97%
  • Reduced Inequalities — Relevance: 73%
DOI: 10.1109/icdmw69685.2025.00069URL

Abstract

Healthcare systems in low- and middle-income countries (LMICs) are overwhelmed and incapable of accommodating increasing demand. Millions of people turn to self-medication without proper guidance, which often results in delayed diagnosis and adverse health outcomes. Many common medications then lose their effectiveness due to improper use. Such an environment creates a critical need for accessible and reliable AI-driven systems to guide patients toward appropriate care. To address these challenges, we developed and compared two distinct approaches, an ensemble machine learning (ML) model and a fine-tuned large language model (LLM). To develop the systems, we curated and clinically validated a large-scale dataset of patient cases. Both frameworks were designed to provide personalized health recommendations at the point of care. In a subsequent real-world evaluation against expert physician consensus, the ML model's recommendations achieved 94.6 % accuracy, while the LLM achieved 81 %. Although the ML model demonstrated higher predictive performance, the LLMs showed a stronger understanding of the context in unstructured narratives. These findings provide the foundational evidence for a hybrid health triage system that synergistically combines the narrative interpretation of LLMs with the predictive reliability of ML. Such a system can create a more robust, safe, and trustworthy tool for patient health risk assessment, delivering scalable, clinically validated, and low-cost guidance to significantly enhance affordable healthcare access and improve health equity.

Keywords

  • Ensemble learning
  • large language models
  • low- and middle-income countries
  • self-medication
  • personalized health recommendations
  • health equity
  • hybrid AI models
  • clinical decision support
  • narrative interpretation
  • benchmark dataset

CCDS Authors

Ashraful Islam, PhD
Ashraful Islam

Ashraful Islam, PhD

  • Center Director, CCDS
  • Director, HCI Wing

Assistant Professor

Department of Computer Science and Engineering

Independent University, Bangladesh (IUB)

Published at

CHI, UIST, UbiComp, IJCNN, ICIP, HCII, ICDMW, BLP, IEEE Internet of Things Journal, JMIR, JMIR Human Factors, JMIR Formative Research, IEEE Access, Smart Agricultural Technology, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Human-Computer Interaction, AI for Social Good, AI for Public Health, AI for Impact

Farhana Sarker, PhD
Farhana Sarker, PhD

Farhana Sarker, PhD

  • Director, WiSE Wing

Associate Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

ICDMW, IEEE Access

References

  1. 1.Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, Sebastian Thrun. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639), 115–118[10.1038/nature21056]
  2. 2.Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, Daniel Shu Wei Ting. (2023). Large language models in medicine. Nature Medicine, 29(8), 1930–1940[10.1038/s41591-023-02448-8]
  3. 3.Hannah L Semigran, Jeffrey A Linder, Courtney Gidengil, Ateev Mehrotra. (2015). Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ, 351, h3480[10.1136/bmj.h3480]
  4. 4.Scott Levin, Matthew Toerper, Eric Hamrock, Jeremiah S. Hinson, Sean Barnes, Heather Gardner, Andrea Dugas, Bob Linton, Tom Kirsch, Gabor Kelen. (2017). Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index. Annals of Emergency Medicine, 71(5), 565–574.e2[10.1016/j.annemergmed.2017.08.005]
  5. 5.Rui Yang, Ting Fang Tan, Wei Lu, Arun James Thirunavukarasu, Daniel Shu Wei Ting, Nan Liu. (2023). Large language models in health care: Development, applications, and challenges. Health care science, 2(4), 255–263[10.1002/hcs2.61]
  6. 6.Tadeusz Ciecierski-Holmes, Ritvij Singh, Miriam Axt, Stephan Brenner, Sandra Barteit. (2022). Artificial intelligence for strengthening healthcare systems in low- and middle-income countries: a systematic scoping review. npj Digital Medicine, 5(1), 162[10.1038/s41746-022-00700-y]
  7. 7.Mohammed Badawy, Nagy Ramadan, Hesham Ahmed Hefny. (2023). Healthcare predictive analytics using machine learning and deep learning techniques: a survey. Journal of Electrical Systems and Information Technology, 10(1)[10.1186/s43067-023-00108-y]
  8. 8.Ziyao Zhang, Chong Wang, Yanlin Wang, Ensheng Shi, Yuchi Ma, Wanjun Zhong, Jiachi Chen, Mingzhi Mao, Zibin Zheng. (2025). LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation. Proceedings of the ACM on software engineering., 2(ISSTA), 481–503[10.1145/3728894]
  9. 9.Jean-Christophe Bélisle-Pipon. (2024). Why we need to be careful with LLMs in medicine. Frontiers in Medicine, 11, 1495582[10.3389/fmed.2024.1495582]
  10. 10.Jonathan Ilicki. (2022). Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review. PLoS ONE, 17(12), e0279636[10.1371/journal.pone.0279636]
  11. 11.Cyrielle Brossard, Christophe Goetz, Pierre Catoire, Lauriane Cipolat, Christophe Guyeux, Cédric Gil Jardine, Mahuna Akplogan, Laure Abensur Vuillaume. (2025). Predicting emergency department admissions using a machine-learning algorithm: a proof of concept with retrospective study. BMC Emergency Medicine, 25(1), 3[10.1186/s12873-024-01141-4]
  12. 12.Audrey Mumbi, Peter Mugo, Edwine Barasa, Gilbert Abotisem Abiiro, Jacinta Nzinga. (2024). Factors influencing the uptake of public health interventions delivery by community pharmacists: A systematic review of global evidence. PLoS ONE, 19(8), e0298713[10.1371/journal.pone.0298713]
  13. 13.Ali Anaissi, Ali Braytee, Junaid Akram. (2024). Fine-Tuning LLMs for Reliable Medical Question-Answering Services. , 146–153[10.1109/icdmw65004.2024.00026]
  14. 14.Hemang Thakar, Vidisha Pradhan, Jigar Sarda, Biswajit Brahma, Akash Kumar Bhoi. (2025). Case Study to Role of Large Language Models in Prediction of the Future Illness. Studies in computational intelligence, 275–310[10.1007/978-3-031-82516-3_13]
  15. 15.Ferdous Wahid Anik, Fahmid Al Rifat, Marzia Zaman, Farhana Sarkar, Khondakar A. Mamun. (2024). The Importance of Prescription Audits: Navigating Complexities and AI-Driven Advancement. , 240–245[10.1109/becithcon64160.2024.10962770]
  16. 16.Swapnil Biswas, Syed Muhammad Mahdi Raza, Thien Huu Nguyen, Robert W. LeAnder, Scott E. Umbaugh. (2025). Advancing skin lesion classification: the role of SAM-based segmentation in enhancing convolutional neural network performance. , 11[10.1117/12.3065735]
← All publications