Conference Paper2019Cited by 2

Intrinsic evaluation of bangla word embeddings

Nafiz Sadman, Akib Sadmanee, Md Iftekhar Tanveer, Md Ashraful Amin, Amin Ahsan Ali

2019 International Conference on Bangla Speech and Language Processing (ICBSLP)

IEEE, pp. 1–5

DOI: 10.1109/icbslp47725.2019.201506URL

Abstract

Word embeddings are vector representations of word that allow machines to learn semantic and syntactic meanings by performing computations on them. Two wellknown embedding models are CBOW and Skipgram. Different methods proposed to evaluate the quality of embeddings are categorized into extrinsic and intrinsic evaluation methods. This paper focuses on intrinsic evaluation - the evaluation of the models on tasks, such as analogy prediction, semantic relatedness, synonym detection, antonym detection and concept categorization. We present intrinsic evaluations on Bangla word embedding created using CBOW and Skipgram models on a Bangla corpus that we built. These are trained on more than 700,000 articles consisting of more than 1.3 million unique words with different embedding dimension sizes, e.g., 300, 100, 64, and 32. We created the evaluation datasets for the abovementioned tasks and performed a comprehensive evaluation. We observe, word vectors of dimension 300, produced using Skipgram models, achieves accuracy of 51.33% for analogy prediction, a correlation of 0.62 for semantic relatedness, and accuracy of 53.85% and 9.56% for synonym and antonym detection 9.56%. Finally, for concept categorization the accuracy is 91.02%. The corpus and evaluation datasets are made publicly available for further research.

Keywords

  • semantic relatedness
  • concept classification
  • synonym detection
  • CBOW
  • intrinsic evaluation
  • skip-gram

CCDS Authors

Prof. M Ashraful Amin, PhD
M Ashraful Amin, PhD

Prof. M Ashraful Amin, PhD

  • Founder Director, CCDS
  • Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Published at

ACL, CHI, UbiComp, ECAI, WACV, PAKDD, IJCNN, ICPR, ICIP, HCII, CLEF, BLP, Scientific Reports, JMIR, IEEE Access, Smart Agricultural Technology, Frontiers in Computational Neuroscience, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

Amin Ahsan Ali, PhD
Amin Ahsan Ali, PhD

Amin Ahsan Ali, PhD

  • Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

ACL, CHI, UbiComp, ECCV, IJCAI, ECAI, WACV, IPSN, PAKDD, LREC, IJCNN, ICPR, ICIP, BEA, BLP, ICMLA, Scientific Reports, Scientific Data, Pattern Recognition, Information Sciences, Ad Hoc Networks, Drug and Alcohol Dependence, PLOS ONE

Research Interests

Artificial Intelligence, Machine Learning

References

  1. 1.Zellig S. Harris. (1954). Distributional Structure. WORD, 10(2-3), 146–162[10.1080/00437956.1954.11659520]
  2. 2.Ping Li, Trevor J. Hastie, Kenneth W. Church. (2006). Very sparse random projections. , 287–296[10.1145/1150402.1150436]
  3. 3.Tobias Schnabel, Igor Labutov, David Mimno, Thorsten Joachims. (2015). Evaluation methods for unsupervised word embeddings. , 298–307[10.18653/v1/d15-1036]
  4. 4.(1936). The Psycho-Biology of Language. Notes and Queries, CLXX(mar28), 234[10.1093/nq/clxx.mar28.234b]
  5. 5.Zellig S. Harris. (1970). Distributional Structure. , 775–794[10.1007/978-94-017-6059-1_36]
  6. 6.Rémi Lebret, Ronan Collobert. (2014). Word Embeddings through Hellinger PCA. , 482–490[10.3115/v1/e14-1051]
  7. 7.Md. Al-Amin, Md. Saiful Islam, Shapan Das Uzzal. (2017). Sentiment analysis of Bengali comments with Word2Vec and sentiment information of words. , 186–190[10.1109/ecace.2017.7912903]
  8. 8.Adnan Ahmad, Mohammad Ruhul Amin. (2016). Bengali word embeddings and it's application in solving document classification problem. , 425–430[10.1109/iccitechn.2016.7860236]
  9. 9.Sakhawat Hosain Sumit, Md. Zakir Hossan, Tareq Al Muntasir, Tanvir Sourov. (2018). Exploring Word Embedding for Bangla Sentiment Analysis. , 1–5[10.1109/icbslp.2018.8554443]
  10. 10.Zakia Sultana Ritu, Nafisa Nowshin, Md Mahadi Hasan Nahid, Sabir Ismail. (2018). Performance Analysis of Different Word Embedding Models on Bangla Language. , 1–5[10.1109/icbslp.2018.8554681]
  11. 11.Hemayet Ahmed Chowdhury, Md. Azizul Haque Imon, Md. Saiful Islam. (2018). A Comparative Analysis of Word Embedding Representations in Authorship Attribution of Bengali Literature. , 1–6[10.1109/iccitechn.2018.8631977]
  12. 12.P. V. Veena, M. Anand Kumar, K. P. Soman. (2017). An effective way of word-level language identification for code-mixed facebook comments using word-embedding via character-embedding. , 1552–1556[10.1109/icacci.2017.8126062]
  13. 13.Redwanul Karim, M. A. Muhiminul Islam, Sazid Rahman Simanto, Saif Ahmed Chowdhury, Kalyan Roy, Adnan Al Neon, Md. Sajid Hasan, Adnan Firoze, Rashedur M. Rahman. (2019). A step towards information extraction: Named entity recognition in Bangla using deep learning. Journal of Intelligent & Fuzzy Systems, 37(6), 7401–7413[10.3233/jifs-179349]
  14. 14.Md Shajalal, Masaki Aono. (2018). Semantic Textual Similarity in Bengali Text. , 1–5[10.1109/icbslp.2018.8554940]
  15. 15.Md. Rajib Hossain, Mohammed Moshiul Hoque. (2018). Automatic Bengali Document Categorization Based on Word Embedding and Statistical Learning Approaches. 2018 International Conference on Computer, Communication, Chemical, Material and Electronic Engineering (IC4ME2), 1–6[10.1109/ic4me2.2018.8465632]
  16. 16.Md. Nowshad Hasan, Sourav Bhowmik, Md. Mahfuzur Rahaman. (2017). Multi-label sentence classification using Bengali word embedding model. 2017 3rd International Conference on Electrical Information and Communication Technology (EICT), 1–6[10.1109/eict.2017.8275207]
  17. 17.Mikolov, Tomas, Chen, Kai, Corrado, Greg, Dean, Jeffrey. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv (Cornell University)[10.48550/arxiv.1301.3781]
  18. 18.Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg, Dean, Jeffrey. (2013). Distributed Representations of Words and Phrases and their Compositionality. arXiv (Cornell University), 26, 3111–3119[10.48550/arxiv.1310.4546]
  19. 19.Dhillon, Paramveer, Rodu, Jordan, Foster, Dean, Ungar, Lyle. (2012). Two Step CCA: A new spectral method for estimating vector models of words. arXiv (Cornell University), 67–74[10.48550/arxiv.1206.6403]

Cited By

  1. 1.Md. Rajib Hossain, Mohammed Moshiul Hoque. (2020). Towards Bengali Word Embedding: Corpus Creation, Intrinsic and Extrinsic Evaluations. Preprints.org[10.20944/preprints202012.0600.v1]
  2. 2.Aminul Islam Bulbul, Saurav Das, Sheikh Abu Haidar Noori. (2024). A Comprehensive Study of Bengali Embedding Models: Insights and Evaluations. , 100–111[10.23919/fruct64283.2024.10749935]
← All publications