Siam Tahsin Bhuiyan
- Intern, CCDS
Bachelors
Independent University, Bangladesh (IUB)
Published at
Scientific Reports
Book Chapter2027
A Multilingual Deep Learning Framework for Sign Language Recognition and Translation
Sharmin Islam Shroddha, Sumaia Anjum Shaba, Sudipto Biswas, Siam Tahsin Bhuiyan, Rashedur Rahman, Saadia Binte Alam, Ashraful Islam
Proceedings of the Seventh International Conference on Trends in Computational and Cognitive Engineering
Springer Nature Switzerland, pp. 115-124, ISBN: 9783032338044
Abstract
Hearing and speech impairments affect millions worldwide, often limiting participation in education, healthcare, and social activities. Sign language is the primary communication mode for these communities; however, the lack of a universal standard and limited translation systems present barriers to inclusion. This paper proposes a multilingual sign language recognition and translation frame-work supporting Bangla (BdSL), Indian (ISL), Arabic (ArSL), and American Sign Language (ASL). The system employs four state-of-the-art deep learning models, VGG16, ResNet50, Vision Transformer (ViT), and EfficientNetB0 to classify sign language images into words. Recognized text is translated into target languages using Google Translate, with the additional capability of regenerating target sign language gestures. Experimental results show all models performed effectively, with ResNet50 and VGG16 achieving 96% accuracy, EfficientNet-B0 94%, and ViT 93%, and all models obtaining an AUC of 0.99. The frame-work demonstrates strong recognition performance, scalability, and potential for cross-lingual communication.
Keywords
- sign language recognition
- sign language translation
- Bangla Sign Language
- Indian Sign Language
- Arabic Sign Language
- American Sign Language
- image classification
- interlingual communication
- deaf and hard of hearing
- vision transformer
- ResNet50
- VGG16
- EfficientNetB0
CCDS Authors
Md Rashedur Rahman, D. Eng.
- Co-Director, MIRA & NEST Wings
Assistant Professor
Department of Computer Science and Engineering
Independent University, Bangladesh (IUB)
Published at
CHI, HCII, Scientific Reports, Cognitive Computation, Neural Computing and Applications
Saadia Binte Alam, PhD
- Director, MIRA & WiSE Wings
Associate Professor
Department of Computer Science and Engineering
Independent University, Bangladesh (IUB)
Published at
HCII, Scientific Reports, Cognitive Computation, Neural Computing and Applications
Ashraful Islam, PhD
- Center Director, CCDS
- Director, HCI Wing
Assistant Professor
Department of Computer Science and Engineering
Independent University, Bangladesh (IUB)
Published at
CHI, UIST, UbiComp, IJCNN, ICIP, HCII, ICDMW, BLP, Pattern Recognition, IEEE Internet of Things Journal, JMIR, JMIR Human Factors, JMIR Formative Research, IEEE Access, Smart Agricultural Technology, Diabetes Research and Clinical Practice, PLOS ONE
References
- 1.Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. (2016). Deep Residual Learning for Image Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 770–778[10.1109/cvpr.2016.90]
- 2.Sunanda Das, Md. Samir Imtiaz, Nieb Hasan Neom, Nazmul Siddique, Hui Wang. (2022). A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier. Expert Systems with Applications, 213, 118914[10.1016/j.eswa.2022.118914]
- 3.M. M. Kamruzzaman. (2020). Arabic Sign Language Recognition and Generating Arabic Speech Using Convolutional Neural Network. Wireless Communications and Mobile Computing, 2020, 1–9[10.1155/2020/3685614]
- 4.Yaser Saleh, Ghassan Farid Issa. (2020). Arabic Sign Language Recognition through Deep Neural Networks Fine-Tuning. International Journal of Online and Biomedical Engineering (iJOE), 16(05), 71–83[10.3991/ijoe.v16i05.13087]
- 5.Jiahui Tao, Yuehan Gu, JiaZheng Sun, Yuxuan Bie, Hui Wang. (2021). Research on vgg16 convolutional neural network feature classification algorithm based on Transfer Learning. China International SAR Symposium (CISS), 1–3[10.23919/ciss51089.2021.9652277]
- 6.Shartaz Khan Akash, Debobrata Chakraborty, Mehedi Mahmud Kaushik, Barsan Saha Babu, Md. Saniat Rahman Zishan. (2023). Action Recognition Based Real-time Bangla Sign Language Detection and Sentence Formation. International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST), 311–315[10.1109/icrest57604.2023.10070072]
- 7.Md Hadiuzzaman, Mohammed Ali, Abu Saleh Miah, Abduur Raj Shafi, Jungpil Shin. (2024). BAUST Lipi: A BdSL Dataset with Deep Learning Based Bangla Sign Language Recognition. International Conference on Computing Advancements (ICCA), 280–285[10.1145/3723178.3723215]
- 8.Mainul Karim, Maun Ali, Md. Ferdous Hasan, Most Fatema-Tuj-Jahra, Syeda Sumaia Sultana, Dewan Md. Farid. (2023). Bangla Sign Language Recognition using YOLOv5. IEEE International Conference for Convergence in Technology (I2CT), 1–7[10.1109/i2ct57861.2023.10126419]
- 9.Sumaia Anjum Shaba, Sharmin Islam Shroddha, Md Junayed Hossain, Md Fahad Monir. (2024). NeuralGesture Communication: Translating one Sign Language to Another Sign Language Using Deep Learning Model and gTTs. IEEE Vehicular Technology Conference, 1–5[10.1109/vtc2024-spring62846.2024.10683444]
- 10.Dosovitskiy, Alexey, Beyer, Lucas, Kolesnikov, Alexander, Weissenborn, Dirk, Zhai, Xiaohua, Unterthiner, Thomas, Dehghani, Mostafa, Minderer, Matthias, Heigold, Georg, Gelly, Sylvain, Uszkoreit, Jakob, Houlsby, Neil. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv (Cornell University)[10.48550/arxiv.2010.11929]
- 11.Tan, Mingxing, Le, Quoc V.. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. arXiv (Cornell University)[10.48550/arxiv.1905.11946]
- 12.Al-Barham, Muhammad, Alsharkawi, Adham, Al-Yaman, Musa, Al-Fetyani, Mohammad, Elnagar, Ashraf, SaAleek, Ahmad Abu, Al-Odat, Mohammad. (2023). RGB Arabic Alphabets Sign Language Dataset. arXiv (Cornell University)[10.48550/arxiv.2301.11932]