Conference Paper2024Cited by 8

Recognizing Medicine Names from Bangladeshi Handwritten Prescription Images Using TrOCR

Mohammad Faiyaz Uz Zaman, Bushra Rahman, Afsana Rubyat, Nabarun Halder, Asif Mahmud, Ashraful Islam, M. Ashraful Amin

2024 7th Asia Conference on Cognitive Engineering and Intelligent Interaction (CEII 2024)

IEEE, pp. 1-5

  • Good Health and Well-being — Relevance: 89%
DOI: 10.1109/ceii65291.2024.00020URL

Abstract

Handwritten prescriptions are widely used in almost every healthcare facility which imposes a major drawback. These prescriptions are often difficult to understand, resulting in the wrong interpretation of medication information, leaving the health of patients at risk. This research study illustrates methods to extract medication information from handwritten prescriptions utilizing a straightforward, deep learning model called TrOCR: Transformer-Based Optical Character Recognition. This pre-trained model can detect complex handwritten language accurately by integrating the Optical Character Recognition (OCR) technique with Transformer architecture. In this study, we utilized a publicly available dataset containing 893 annotated prescription images to train and test the model. The OCR performance of our trained model on the test set showed a Character Error Rate (CER) of 16% and a Word Error Rate (WER) of 36%. With the integration of the Levenshtein distance-based text correction technique, the CER and WER dropped significantly to 7% and 11%, indicating improvement in the recognition of medication information.

Keywords

  • Word error rate
  • TrOCR
  • character error rate

CCDS Authors

Nabarun Halder
Nabarun Halder

Nabarun Halder

  • Research Assistant, CCDS

Published at

UbiComp, IJCNN, HCII, IEEE Access

Research Interests

Machine Learning, Deep Learning, Natural Language Processing, Health Informatics, Human-Computer Interaction

Asif Mahmud
Asif Mahmud

Asif Mahmud

  • Co-Director, IPT Wing

Lecturer

Department of Computer Science and Engineering

Independent University, Bangladesh

Ashraful Islam, PhD
Ashraful Islam

Ashraful Islam, PhD

  • Center Director, CCDS
  • Director, HCI Wing

Assistant Professor

Department of Computer Science and Engineering

Independent University, Bangladesh (IUB)

Published at

CHI, UIST, UbiComp, IJCNN, ICIP, HCII, ICDMW, BLP, IEEE Internet of Things Journal, JMIR, JMIR Human Factors, JMIR Formative Research, IEEE Access, Smart Agricultural Technology, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Human-Computer Interaction, AI for Social Good, AI for Public Health, AI for Impact

Prof. M Ashraful Amin, PhD
M Ashraful Amin, PhD

Prof. M Ashraful Amin, PhD

  • Founder Director, CCDS
  • Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Published at

ACL, CHI, UbiComp, ECAI, WACV, PAKDD, IJCNN, ICPR, ICIP, HCII, CLEF, BLP, Scientific Reports, JMIR, IEEE Access, Smart Agricultural Technology, Frontiers in Computational Neuroscience, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

References

  1. 1.Jakubův, Jan, Chvalovský, Karel, Goertzel, Zarathustra, Kaliszyk, Cezary, Olšák, Mirek, Piotrowski, Bartosz, Schulz, Stephan, Suda, Martin, Urban, Josef. (2023). MizAR 60 for Mizar 50. DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)[10.4230/lipics.itp.2023.19]
  2. 2.Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florêncio, Cha Zhang, Zhoujun Li, Furu Wei. (2023). TrOCR: Transformer-Based Optical Character Recognition with Pre-trained Models. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 13094–13102[10.1609/aaai.v37i11.26538]
  3. 3.Dietrich Klakow, Jochen Peters. (2002). Testing the correlation of word error rate and perplexity. Speech Communication, 38(1-2), 19–28[10.1016/s0167-6393(01)00041-3]
  4. 4.Harald Scheidl, Stefan Fiel, Robert Sablatnig. (2018). Word Beam Search: A Connectionist Temporal Classification Decoding Algorithm. , 253–258[10.1109/icfhr-2018.2018.00052]
  5. 5.Musa Dildar Ahmed Cheema, Mohammad Daniyal Shaiq, Farhaan Mirza, Ali Kamal, M. Asif Naeem. (2024). Adapting multilingual vision language transformers for low-resource Urdu optical character recognition (OCR). PeerJ Computer Science, 10, e1964[10.7717/peerj-cs.1964]
  6. 6.Jasper Beernaerts, Ellen Debever, Matthieu Lenoir, Bernard De Baets, Nico Van de Weghe. (2018). A method based on the Levenshtein distance metric for the comparison of multiple movement patterns described by matrix sequences of different length. Expert Systems with Applications, 115, 373–385[10.1016/j.eswa.2018.07.076]
  7. 7.A. Nithyakalyani, S. Jothilakshmi. (2019). Speech Summarization for Tamil Language. Elsevier eBooks, 113–138[10.1016/b978-0-12-818130-0.00007-6]
  8. 8.Julinda Gllavata, Ralph Ewerth, Bernd Freisleben. (2005). A Text Detection, Localization and Segmentation System for OCR in Images. , 310–317[10.1109/mmse.2004.18]

Cited By

  1. 1.Jeevana Jyothi Pujari, Thulasi Bikku, Kranthi Kumar Singamaneni, A. C. Priya Ranjani, Nasr Al Din Ide, Srinivasarao Thota. (2026). A hybrid framework for pill identification using convolutional neural networks and optical character recognition. Discover Artificial Intelligence, 6(1)[10.1007/s44163-026-01405-x]
  2. 2.Fatima Bukhari, Mickael Coustaty, Muhammad Muzzamil Luqman, Souhail Bakkali, Michael G. Madden, Nadeem Iqbal Kajla. (2026). BHCP-DOC: A Real-World Bilingual Dataset for Handwritten Clinical Prescription Detection and Recognition. Lecture notes in computer science, 448–463[10.1007/978-3-032-39379-1_31]
  3. 3.A. Sai Sachin, J. Karthikeyan. (2026). A lightweight hybrid deep learning framework for multi-pill detection, multi-attribute recognition, OCR-based imprint analysis, and metadata retrieval. Frontiers in Artificial Intelligence, 9, 1890296[10.3389/frai.2026.1890296]
  4. 4.Sai Chaitanya Tummala, Manoj Yenigalla, Venkatrama Phani Kumar S, Venkata Krishna Kishore K. (2026). An Automated Medical Strip Recognition and Billing System using Deep Learning Techniques. International Conference on Sustainable Computing and Smart Systems (ICSCSS), 956–963[10.1109/icscss69635.2026.11645893]
  5. 5.R. M. Gomathi, Satyavarapu Naga Siva Uttej, Shaik Mohammad Afzal, Senduru Srinivasulu, T. Anandhi, P. Malathi. (2026). Intelligent Drug Inventory Prediction Using OCR and Machine Learning. , 1246–1251[10.1109/icsscna68616.2026.11546725]
  6. 6.Abdulla Al Mobassir, Md. Rayhan Hasan, Mst. Afsana Sadia Borni, Md. Razu Ahmed, Rumman Ahmed Prodhan. (2026). Multimodal Deep Learning for Handwritten Prescription Recognition with Explainable AI. , 1–6[10.1109/qpain69676.2026.11546070]
  7. 7.Ajit Kumar Samal, N. Toyaad Kumar Reddy, Brojo Kishore Mishra. (2026). AutoRxAnnotator: A Multimodal Deep Learning Framework for Automated Annotation and Labeling of Handwritten and Digital Medical Prescriptions. , 1–6[10.1109/ic-icns68863.2026.11537514]
  8. 8.Chuanjun Chen, Yiru Yin, Junjie Liu. (2025). Knowledge-Enhanced TrOCR for Automated Storage and Retrieval Systems: A Novel Approach to Pharmaceutical Package Text Recognition. , 27–31[10.1109/icairc68035.2025.11385200]
← All publications