Conference Paper2026

Real-Time Bangla Sign Language Interpretation Using Vision Transformers and Micro-Gesture Analysis

Jahanggir Hossain Setu, Anika Jasim Ema, Mst. Aysa Siddika Meem, Ashraful Islam, Rashedur Rahman, M Ashraful Amin

2026 International Conference on Advanced Computing Technologies (ICACT)

IEEE, pp. 1-6

Abstract

Sign language plays an essential role in communication for individuals with hearing and speech disabilities. Each spoken language has a corresponding sign language that translates visual gestures into written form. This study addresses the need for effective sign language interpretation tools by developing a web application for Bangla Sign Language (BSL). The study uses Deep Learning (DL) models to improve communication between sign language users and non-users. The application integrates Google Mediapipe for accurate pose estimation and a Vision Transformer (ViT) model for micro-gesture classification. The use of pose estimation guarantees unbiased performance across different skin tones by focusing on anatomical structures rather than raw pixel inputs to the ViT. The ViT model demonstrated high-performance metrics, with an accuracy of 97.41%, precision of 96.83%, recall of 95.98%, and an F1-Score of 96.4%. Additionally, the Google Translator Application Programming Interface (API) translates the generated Bangla text into multiple languages which enhances cross-cultural communication. A Text-To-Speech (TTS) feature converts Bangla text into spoken language, addressing communication barriers for visually impaired individuals. By overcoming the limitations of traditional alphabet-based interpretation methods, the developed web application provides a practical solution for real-time BSL translation. This study contributes to improving accessibility and inclusivity for the deaf and hearing-impaired community.

Keywords

  • Bangla Sign Language
  • sign language recognition
  • vision transformer
  • pose estimation
  • Web application
  • text-to-speech
  • Google Translate API
  • Accessibility

CCDS Authors

Ashraful Islam, PhD
Ashraful Islam

Ashraful Islam, PhD

  • Center Director, CCDS
  • Director, HCI Wing

Published at

CHI, UIST, UbiComp, IJCNN, ICIP, HCII, ICDMW, BLP, ICMLA, Scientific Data, Pattern Recognition, IEEE Internet of Things Journal, JMIR, JMIR Human Factors, JMIR Formative Research, IEEE Access, Smart Agricultural Technology, Diabetes Research and Clinical Practice, PLOS ONE

Md Rashedur Rahman, D. Eng.
Md. Rashedur Rahman, D. Eng.

Md Rashedur Rahman, D. Eng.

  • Co-Director, MIRA & NEST Wings
  • Assistant Professor

    Department of Computer Science and Engineering

    Independent University, Bangladesh (IUB)

Published at

CHI, HCII, Scientific Reports, Scientific Data, Cognitive Computation, Neural Computing and Applications

Prof. M Ashraful Amin, PhD
M Ashraful Amin, PhD

Prof. M Ashraful Amin, PhD

  • Founder Director, CCDS
  • Director, AI/ML, HCI & DS Wings

Published at

ACL, CHI, UbiComp, ECAI, WACV, PAKDD, LREC, IJCNN, ICPR, ICIP, HCII, CLEF, BLP, ICMLA, Scientific Reports, Pattern Recognition, JMIR, IEEE Access, Smart Agricultural Technology, Frontiers in Computational Neuroscience, Diabetes Research and Clinical Practice, PLOS ONE

References

  1. 1.Mehta, Sachin, Rastegari, Mohammad. (2023). Separable Self-attention for Mobile Vision Transformers. Transactions on machine learning research[10.48550/arxiv.2206.02680]
  2. 2.Kshitij Bantupalli, Ying Xie. (2018). American Sign Language Recognition using Deep Learning and Computer Vision. IEEE International Conference on Big Data (BigData), 4896–4899[10.1109/bigdata.2018.8622141]
  3. 3.Jong-Wook Kim, Jin-Young Choi, Eun-Ju Ha, Jae-Ho Choi. (2023). Human Pose Estimation Using MediaPipe Pose and Optimization Method Based on a Humanoid Model. Applied Sciences, 13(4), 2700[10.3390/app13042700]
  4. 4.Cosmin Munteanu, Heather Molyneaux, Wendy Moncur, Mario Romero, Susan O'Donnell, John Vines. (2015). Situational Ethics: Re-thinking Approaches to Formal Ethics Requirements for Human-Computer Interaction. CHI Conference on Human Factors in Computing Systems, 105–114[10.1145/2702123.2702481]
  5. 5.Abu Saleh Musa Miah, Jungpil Shin, Md Al Mehedi Hasan, Md Abdur Rahim. (2022). BenSignNet: Bengali Sign Language Alphabet Recognition Using Concatenated Segmentation and Convolutional Neural Network. Applied Sciences, 12(8), 3933[10.3390/app12083933]
  6. 6.Sumaya Siddique, Shafinul Islam, Emon Emtiyaz Neon, Tajnoor Sabbir, Intisar Tahmid Naheen, Riasat Khan. (2023). Deep Learning-based Bangla Sign Language Detection with an Edge Device. Intelligent Systems with Applications, 18, 200224[10.1016/j.iswa.2023.200224]
  7. 7.Abdul Muntakim Rafi, Nowshin Nawal, Nur Sultan Nazar Bayev, Lusain Nima, Celia Shahnaz, Shaikh Anowarul Fattah. (2019). Image-based Bengali Sign Language Alphabet Recognition for Deaf and Dumb Community. IEEE Global Humanitarian Technology Conference (GHTC), 1–7[10.1109/ghtc46095.2019.9033031]
  8. 8.Nanziba Basnin, Lutfun Nahar, Mohammad Shahadat Hossain. (2020). An Integrated CNN-LSTM Model for Bangla Lexical Sign Language Recognition. Advances in intelligent systems and computing, 695–707[10.1007/978-981-33-4673-4_57]
  9. 9.Ragib Amin Nihal, Sejuti Rahman, Nawara Mahmood Broti, Shamim Ahmed Deowan. (2021). Bangla Sign alphabet recognition with zero-shot and transfer learning. Pattern Recognition Letters, 150, 84–93[10.1016/j.patrec.2021.06.020]
  10. 10.Tazkia Mim Angona, A. S. M. Siamuzzaman Shaon, Kazi Tahmid Rashad Niloy, Tajbia Karim, Zarin Tasnim, S. M. Salim Reza, Tasmima Noushiba Mahbub. (2020). Automated Bangla sign language translation system for alphabets by means of MobileNet. TELKOMNIKA (Telecommunication Computing Electronics and Control), 18(3), 1292[10.12928/telkomnika.v18i3.15311]
  11. 11.Thasin Abedin, Khondokar S. S. Prottoy, Ayana Moshruba, Safayat Bin Hakim. (2023). Bangla Sign Language Recognition Using a Concatenated BdSL Network. Chapman & Hall/CRC eBooks, 76–86[10.1201/9781003256106-7]
  12. 12.Dardina Tasmere, Boshir Ahmed. (2020). Hand Gesture Recognition for Bangla Sign Language Using Deep Convolution Neural Network. International Conference on Sustainable Technologies for Industry 5.0 (STI), 1–5[10.1109/sti50764.2020.9350484]
  13. 13.Supriya Krishnamurthi, Indiramma M. (2021). Sign Language Translator Using Deep Learning Techniques. International Conference on Electrical, Computer and Communication Technologies (ICECCT), 1–5[10.1109/icecct52121.2021.9616795]
  14. 14.Md. Mahmudul Hasan, Sk. Mohammad Masudul Ahsan. (2019). Bangla Sign Digits Recognition Using HOG Feature Based Multi-Class Support Vector Machine. International Conference on Electrical Information and Communication Technology (EICT), 1–5[10.1109/eict48899.2019.9068832]
  15. 15.Prosenjit Roy, S.M Miraj Uddin, Md. Arifur Rahman, Md. Musfiqur Rahman, Md. Shahin Alam, Md. Saidur Rashid Mahin. (2019). Bangla Sign Language Conversation Interpreter Using Image Processing. International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), 1–5[10.1109/icasert.2019.8934614]
  16. 16.Nasima Begum, Saqib Sizan Khan, Rashik Rahman, Ashraful Haque, Nipa Khatun, Nusrat Jahan, Tanjina Helaly. (2022). QMX-BdSL49: An Efficient Recognition Approach for Bengali Sign Language with Quantize Modified Xception. International Journal of Advanced Computer Science and Applications, 14(5)[10.14569/ijacsa.2023.01405113]
  17. 17.Riitta Lahtinen, Stina Ojala. (2018). Interpreting Behaviour and Emotions for People with Deafblindness. Communications in computer and information science, 211–220[10.1007/978-3-319-97931-1_17]