Conference Paper2023Cited by 1

Contextual Bangla Neural Stemmer: Finding Contextualized Root-Word Representations for Bangla Words

M. Fahim, A. A. Ali, M. A. Amin, A. K. M. M. Rahman

Workshop on Bangla Language Processing in conjunction with EMNLP

Association for Computational Linguistics, pp. 94–103

DOI: 10.18653/v1/2023.banglalp-1.11Open AccessURL

Abstract

Stemmers are commonly used in NLP to reduce words to their root form. However, this process may discard important information and yield incorrect root forms, affecting the accuracy of NLP tasks. To address these limitations, we propose a Contextual Bangla Neural Stemmer for Bangla language to enhance word representations. Our method involves splitting words into characters within the Neural Stemming Block, obtaining vector representations for both stem words and unknown vocabulary words. A loss function aligns these representations with Word2Vec representations, followed by contextual word representations from a Universal Transformer encoder. Mean Pooling generates sentence-level representations that are aligned with BanglaBERT’s representations using a MLP layer. The proposed model also tries to build good representations for out-of-vocabulary (OOV) words. Experiments with our model on five Bangla datasets shows around 5% average improvement over the vanilla approach. Notably, our method avoids BERT retraining, focusing on root word detection and addressing OOV and sub-word issues. By incorporating our approach into a large corpus-based Language Model, we expect further improvements in aspects like explainability.

Keywords

  • contextualized word representations
  • out-of-vocabulary words
  • Bengali language
  • sentence-level representation

References

  1. 1.Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. (2019). , 4171–4186[10.18653/v1/n19-1423]
  2. 2.Rivo Kaugeranna, Eliina Kaugeranna, Aion, (Claude Sonnet 4.6). (2023). Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics. DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)[10.4230/lipics.giscience.2023.43]
  3. 3.Abhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, Kazi Samin Mubasshir, Md Saiful Islam, Anindya Iqbal, M. Sohel Rahman, Rifat Shahriyar. (2022). BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla. Findings of the Association for Computational Linguistics: NAACL 2022, 1318–1327[10.18653/v1/2022.findings-naacl.98]
  4. 4.Khondoker Ittehadul Islam, Sudipta Kar, Md Saiful Islam, Mohammad Ruhul Amin. (2021). SentNoB: A Dataset for Analysing Sentiment on Noisy Bangla Texts. , 3265–3271[10.18653/v1/2021.findings-emnlp.278]
  5. 5.Abhisek Chakrabarty, Onkar Arun Pandit, Utpal Garain. (2017). Context Sensitive Lemmatization Using Two Successive Bidirectional Gated Recurrent Networks. , 1481–1491[10.18653/v1/p17-1136]
  6. 6.Abhisek Chakrabarty, Utpal Garain. (2016). BenLem (A Bengali Lemmatizer) and Its Role in WSD. ACM Transactions on Asian and Low-Resource Language Information Processing, 15(3), 1–18[10.1145/2835494]
  7. 7.Md. Ashraful Islam, Md. Towhiduzzaman, Md. Tauhidul Islam Bhuiyan, Abdullah Al Maruf, Jesan Ahammed Ovi. (2022). BaNeL: an encoder-decoder based Bangla neural lemmatizer. SN Applied Sciences, 4(5)[10.1007/s42452-022-04985-2]
  8. 8.Abhisek Chakrabarty, Akshay Chaturvedi, Utpal Garain. (2016). A Neural Lemmatizer for Bengali. , 2558–2561[10.63317/2gqaypvup23e]
  9. 9.Souvick Das, Rajat Pandit, Sudip Kumar Naskar. (2020). A Rule Based Lightweight Bengali Stemmer.. International Conference on Networks, 400–408
  10. 10.Loshchilov, Ilya, Hutter, Frank. (2017). Decoupled Weight Decay Regularization. arXiv (Cornell University)[10.48550/arxiv.1711.05101]
  11. 11.Dehghani, Mostafa, Gouws, Stephan, Vinyals, Oriol, Uszkoreit, Jakob, Kaiser, Łukasz. (2018). Universal Transformers. arXiv (Cornell University)[10.48550/arxiv.1807.03819]
  12. 12.Hossain, Md Zobaer, Rahman, Md Ashraful, Islam, Md Saiful, Kar, Sudipta. (2020). BanFakeNews: A Dataset for Detecting Fake News in Bangla. arXiv (Cornell University)[10.48550/arxiv.2004.08789]

Cited By

  1. 1.Farhan Noor Dehan, Md Fahim, A. K. M. Mahabubur Rahman, M. Ashraful Amin, Amin Ahsan Ali. (2024). TinyLLM Efficacy in Low-Resource Language: An Experiment on Bangla Text Classification Task. Lecture notes in computer science, 472–487[10.1007/978-3-031-78495-8_30]
← All publications