Nabarun Halder
- Research Assistant, CCDS
Research Interests
Machine Learning, Deep Learning, Natural Language Processing, Health Informatics, Human-Computer Interaction
Conference Paper2024Cited by 6
Recognizing Medicine Names from Bangladeshi Handwritten Prescription Images Using TrOCR
Mohammad Faiyaz Uz Zaman, Bushra Rahman, Afsana Rubyat, Nabarun Halder, Asif Mahmud, Ashraful Islam, M. Ashraful Amin
2024 7th Asia Conference on Cognitive Engineering and Intelligent Interaction (CEII 2024)
IEEE, pp. 1-5
Abstract
Handwritten prescriptions are widely used in almost every healthcare facility which imposes a major drawback. These prescriptions are often difficult to understand, resulting in the wrong interpretation of medication information, leaving the health of patients at risk. This research study illustrates methods to extract medication information from handwritten prescriptions utilizing a straightforward, deep learning model called TrOCR: Transformer-Based Optical Character Recognition. This pre-trained model can detect complex handwritten language accurately by integrating the Optical Character Recognition (OCR) technique with Transformer architecture. In this study, we utilized a publicly available dataset containing 893 annotated prescription images to train and test the model. The OCR performance of our trained model on the test set showed a Character Error Rate (CER) of 16% and a Word Error Rate (WER) of 36%. With the integration of the Levenshtein distance-based text correction technique, the CER and WER dropped significantly to 7% and 11%, indicating improvement in the recognition of medication information.
Keywords
Medical prescriptionComputer scienceArtificial intelligenceNatural language processingSpeech recognitionPattern recognition (psychology)
CCDS Authors
Asif Mahmud
- Co-Director, IPT Wing
Lecturer
Department of Computer Science and Engineering
Independent University, Bangladesh
Ashraful Islam, PhD
- Center Director, CCDS
- Director, HCI Wing
Assistant Professor
Department of Computer Science and Engineering
Independent University, Bangladesh (IUB)Research Interests
Human-Computer Interaction, AI for Social Good, AI for Public Health, AI for Impact
M Ashraful Amin, PhD
- Founder Director, CCDS
- Director, AI/ML, HCI & DS Wings
Research Interests
Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics
References
- 1.Jakubův, Jan, Chvalovský, Karel, Goertzel, Zarathustra, Kaliszyk, Cezary, Olšák, Mirek, Piotrowski, Bartosz, Schulz, Stephan, Suda, Martin, Urban, Josef. (2023). MizAR 60 for Mizar 50. DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)[10.4230/lipics.itp.2023.19]
- 2.Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florêncio, Cha Zhang, Zhoujun Li, Furu Wei. (2023). TrOCR: Transformer-Based Optical Character Recognition with Pre-trained Models. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 13094–13102[10.1609/aaai.v37i11.26538]
- 3.Dietrich Klakow, Jochen Peters. (2002). Testing the correlation of word error rate and perplexity. Speech Communication, 38(1-2), 19–28[10.1016/s0167-6393(01)00041-3]
- 4.Harald Scheidl, Stefan Fiel, Robert Sablatnig. (2018). Word Beam Search: A Connectionist Temporal Classification Decoding Algorithm. , 253–258[10.1109/icfhr-2018.2018.00052]
- 5.Musa Dildar Ahmed Cheema, Mohammad Daniyal Shaiq, Farhaan Mirza, Ali Kamal, M. Asif Naeem. (2024). Adapting multilingual vision language transformers for low-resource Urdu optical character recognition (OCR). PeerJ Computer Science, 10, e1964[10.7717/peerj-cs.1964]
- 6.Jasper Beernaerts, Ellen Debever, Matthieu Lenoir, Bernard De Baets, Nico Van de Weghe. (2018). A method based on the Levenshtein distance metric for the comparison of multiple movement patterns described by matrix sequences of different length. Expert Systems with Applications, 115, 373–385[10.1016/j.eswa.2018.07.076]
- 7.A. Nithyakalyani, S. Jothilakshmi. (2019). Speech Summarization for Tamil Language. Elsevier eBooks, 113–138[10.1016/b978-0-12-818130-0.00007-6]
- 8.Julinda Gllavata, Ralph Ewerth, Bernd Freisleben. (2005). A Text Detection, Localization and Segmentation System for OCR in Images. , 310–317[10.1109/mmse.2004.18]
Cited By
- 1.Jeevana Jyothi Pujari, Thulasi Bikku, Kranthi Kumar Singamaneni, A. C. Priya Ranjani, Nasr Al Din Ide, Srinivasarao Thota. (2026). A hybrid framework for pill identification using convolutional neural networks and optical character recognition. Discover Artificial Intelligence, 6(1)[10.1007/s44163-026-01405-x]
- 2.R. M. Gomathi, Satyavarapu Naga Siva Uttej, Shaik Mohammad Afzal, Senduru Srinivasulu, T. Anandhi, P. Malathi. (2026). Intelligent Drug Inventory Prediction Using OCR and Machine Learning. , 1246–1251[10.1109/icsscna68616.2026.11546725]
- 3.Abdulla Al Mobassir, Md. Rayhan Hasan, Mst. Afsana Sadia Borni, Md. Razu Ahmed, Rumman Ahmed Prodhan. (2026). Multimodal Deep Learning for Handwritten Prescription Recognition with Explainable AI. , 1–6[10.1109/qpain69676.2026.11546070]
- 4.Chuanjun Chen, Yiru Yin, Junjie Liu. (2025). Knowledge-Enhanced TrOCR for Automated Storage and Retrieval Systems: A Novel Approach to Pharmaceutical Package Text Recognition. , 27–31[10.1109/icairc68035.2025.11385200]
- 5.Ajit Kumar Samal, N. Toyaad Kumar Reddy, Brojo Kishore Mishra. (2026). AutoRxAnnotator: A Multimodal Deep Learning Framework for Automated Annotation and Labeling of Handwritten and Digital Medical Prescriptions. , 1–6[10.1109/ic-icns68863.2026.11537514]
- 6.A. Sai Sachin, J. Karthikeyan. (2026). A lightweight hybrid deep learning framework for multi-pill detection, multi-attribute recognition, OCR-based imprint analysis, and metadata retrieval. Frontiers in Artificial Intelligence, 9, 1890296[10.3389/frai.2026.1890296]