Conference Paper2017Cited by 17

Feature selection and discretization based on mutual information

Sadia Sharmin, Amin Ahsan Ali, Muhammad Asif Hossain Khan, Mohammad Shoyaib

2017 IEEE international conference on imaging, vision & pattern recognition (icIVPR)

IEEE, pp. 1–6

DOI: 10.1109/icivpr.2017.7890885URL

Abstract

Feature selection and discretization have been considered to be an important research topic in the field of pattern recognition and data mining. However, addressing both these issues at a time is rarely discussed in the existing research. In this paper, these issues have been addressed by developing a heuristic namely discretization and selection of features based on mutual information (DSM). Experimental results on 15 datasets show that the proposed DSM outperforms a number of state-of-the-art feature selection or discretization algorithms. On average, its accuracy surpasses that of the best performing state-of-the-art algorithms by 5% using Support Vector Machine. Moreover, for datasets with a large number of features, it shows promising accuracies even with far less number of features than the other competing algorithms.

Keywords

  • Mutual information
  • Feature selection
  • Discretization
  • classification accuracy
  • feature reduction
  • data preprocessing

CCDS Authors

Amin Ahsan Ali, PhD
Amin Ahsan Ali, PhD

Amin Ahsan Ali, PhD

  • Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

ACL, CHI, UbiComp, ECCV, IJCAI, ECAI, WACV, IPSN, PAKDD, LREC, IJCNN, ICPR, ICIP, BEA, BLP, ICMLA, Scientific Reports, Scientific Data, Pattern Recognition, Information Sciences, Ad Hoc Networks, Drug and Alcohol Dependence, PLOS ONE

Research Interests

Artificial Intelligence, Machine Learning

References

  1. 1.Arthur Asuncion. (2007). UCI Machine Learning Repository. Medical Entomology and Zoology
  2. 2.Hanchuan Peng, Fuhui Long, C. Ding. (2005). Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8), 1226–1238[10.1109/tpami.2005.159]
  3. 3.J. Rissanen. (1978). Modeling by shortest data description. Automatica, 14(5), 465–471[10.1016/0005-1098(78)90005-5]
  4. 4.Usama M. Fayyad, Keki B. Irani. (1993). Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning. International Joint Conference on Artificial Intelligence, 2, 1022–1027
  5. 5.R. Battiti. (1994). Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on Neural Networks, 5(4), 537–550[10.1109/72.298224]
  6. 6.Lei Yu, Huan Liu. (2004). Efficient Feature Selection via Analysis of Relevance and Redundancy. , 5, 1205–1224
  7. 7.P.A. Estevez, M. Tesmer, C.A. Perez, J.M. Zurada. (2009). Normalized Mutual Information Feature Selection. IEEE Transactions on Neural Networks, 20(2), 189–201[10.1109/tnn.2008.2005601]
  8. 8.Jorge R. Vergara, Pablo A. Estévez. (2013). A review of feature selection methods based on mutual information. Neural Computing and Applications, 24(1), 175–186[10.1007/s00521-013-1368-0]
  9. 9.Gavin Brown, Adam Pocock, Mingjie Zhao, Mikel Luján. (2012). Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. Research Explorer (The University of Manchester), 13(1), 27–66[link]
  10. 10.Huan Liu, Farhad Hussain, Chew Lim Tan, Manoranjan Dash. (2002). Discretization: An Enabling Technique. Data Mining and Knowledge Discovery, 6(4), 393–423[10.1023/a:1016304305535]
  11. 11.Huan Liu, R. Setiono. (2002). Chi2: feature selection and discretization of numeric attributes. , 388–391[10.1109/tai.1995.479783]
  12. 12.Manoranjan Dash, Huan Liu. (2003). Consistency-based search in feature selection. Artificial Intelligence, 151(1-2), 155–176[10.1016/s0004-3702(03)00079-1]
  13. 13.Gosse Bouma. (2009). Normalized (pointwise) mutual information in collocation extraction.
  14. 14.Mohamed Bennasar, Yulia Hicks, Rossitza Setchi. (2015). Feature selection using Joint Mutual Information Maximisation. Expert Systems with Applications, 42(22), 8520–8532[10.1016/j.eswa.2015.07.007]
  15. 15.Randy Kerber. (1992). ChiMerge: discretization of numeric attributes. National Conference on Artificial Intelligence, 123–128
  16. 16.David D. Lewis. (1992). Feature selection and feature extraction for text categorization. , 212[10.3115/1075527.1075574]
  17. 17.Salvador Garcia, J. Luengo, José Antonio Sáez, Victoria López, F. Herrera. (2012). A Survey of Discretization Techniques: Taxonomy and Empirical Analysis in Supervised Learning. IEEE Transactions on Knowledge and Data Engineering, 25(4), 734–750[10.1109/tkde.2012.35]
  18. 18.L.A. Kurgan, K.J. Cios. (2004). CAIM discretization algorithm. IEEE Transactions on Knowledge and Data Engineering, 16(2), 145–153[10.1109/tkde.2004.1269594]
  19. 19.Shenkai Gu, Ran Cheng, Yaochu Jin. (2016). Feature selection for high-dimensional classification using a competitive swarm optimizer. Soft Computing, 22(3), 811–822[10.1007/s00500-016-2385-6]
  20. 20.Patrick Emmanuel Meyer, Colas Schretter, Gianluca Bontempi. (2008). Information-Theoretic Feature Selection in Microarray Data Using Variable Complementarity. IEEE Journal of Selected Topics in Signal Processing, 2(3), 261–274[10.1109/jstsp.2008.923858]
  21. 21.Cheng-Jung Tsai, Chien-I. Lee, Wei-Pang Yang. (2007). A discretization algorithm based on Class-Attribute Contingency Coefficient. Information Sciences, 178(3), 714–731[10.1016/j.ins.2007.09.004]
  22. 22.Marc Boullé. (2006). MODL: A Bayes optimal discretization method for continuous attributes. Machine Learning, 65(1), 131–165[10.1007/s10994-006-8364-x]
  23. 23.F.E.H. Tay, Lixiang Shen. (2002). A modified Chi2 algorithm for discretization. IEEE Transactions on Knowledge and Data Engineering, 14(3), 666–670[10.1109/tkde.2002.1000349]
  24. 24.J. Bins, B.A. Draper. (2002). Feature selection from huge feature sets. , 2, 159–165[10.1109/iccv.2001.937619]
  25. 25.Kiran S Balagani, Vir V Phoha. (2010). On the Feature Selection Criterion Based on an Approximation of Multidimensional Mutual Information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(7), 1342–1343[10.1109/tpami.2010.62]
  26. 26.Bogdan S. Chlebus, Sinh Hoa Nguyen. (1998). On Finding Optimal Discretizations for Two Attributes. Lecture notes in computer science, 537–544[10.1007/3-540-69115-4_74]
  27. 27.Tofigh Naghibi, Sarah Hoffmann, Beat Pfister. (2014). A Semidefinite Programming Based Search Strategy for Feature Selection with Mutual Information Measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(8), 1529–1541[10.1109/tpami.2014.2372791]
  28. 28.W.-H. Au, K.C.C. Chan, A.K.C. Wong. (2006). A fuzzy approach to partitioning continuous attributes for classification. IEEE Transactions on Knowledge and Data Engineering, 18(5), 715–719[10.1109/tkde.2006.70]
  29. 29.Alberto Cano, Dat T. Nguyen, Sebastián Ventura, Krzysztof J. Cios. (2014). ur-CAIM: improved CAIM discretization for unbalanced and balanced data. Soft Computing, 20(1), 173–188[10.1007/s00500-014-1488-1]
  30. 30.Hongrong Cheng, Zhiguang Qin, Weizhong Qian, Wei Liu. (2008). Conditional Mutual Information Based Feature Selection. , 103–107[10.1109/kam.2008.85]
  31. 31.Deqin Yan, Deshan Liu, Yu Sang. (2014). A new approach for discretizing continuous attributes in learning systems. Neurocomputing, 133, 507–511[10.1016/j.neucom.2013.12.005]
  32. 32.X. Wu. (1996). A Bayesian Discretizer for Real-Valued Attributes. The Computer Journal, 39(8), 688–691[10.1093/comjnl/39.8.688]
  33. 33.João Gama, Luis Torgo, Carlos Soares. (1998). Dynamic Discretization of Continuous Attributes. Lecture notes in computer science, 160–169[10.1007/3-540-49795-1_14]
  34. 34.Gexiang Zhang, Laizhao Hu, Weidong Jin. (2004). Discretization of Continuous Attributes in Rough Set Theory and Its Application. Lecture notes in computer science, 1020–1026[10.1007/978-3-540-30497-5_157]
  35. 35.Artur J. Ferreira, Mário A. T. Figueiredo. (2014). Feature Discretization with Relevance and Mutual Information Criteria. Advances in intelligent systems and computing, 101–118[10.1007/978-3-319-12610-4_7]
  36. 36.Hao-Peng Yan, Wen-Biao Liu. (2016). The third order correction on Hawking radiation and entropy conservation during black hole evaporation process. Physics Letters B, 759, 293–297[10.1016/j.physletb.2016.05.079]
  37. 37.S. A. Fattah, Chia-Chun Lin, Sun-Yuan Kung. (2011). A mutual information based approach for evaluating the quality of clustering. , 22, 601–604[10.1109/icassp.2011.5946475]
  38. 38.S. Partovi, N.E. Nahi. (1969). Absolute stability of dynamic system containing non-linear functions of several state variables. Automatica, 5(4), 465–473[10.1016/0005-1098(69)90108-3]

Cited By

  1. 1.Sadia Sharmin, Mohammad Shoyaib, Amin Ahsan Ali, Muhammad Asif Hossain Khan, Oksam Chae. (2019). Simultaneous feature selection and discretization based on mutual information. Pattern Recognition, 91, 162–174[10.1016/j.patcog.2019.02.016]
  2. 2.Abdul Majeed. (2019). Improving Time Complexity and Accuracy of the Machine Learning Algorithms Through Selection of Highly Weighted Top k Features from Complex Datasets. Annals of Data Science, 6(4), 599–621[10.1007/s40745-019-00217-4]
  3. 3.Md. Nazmul Haque, Mahir Mahbub, Md. Hasan Tarek, Lutfun Nahar Lota, Amin Ahsan Ali. (2019). Nurse care activity recognition. , 719–723[10.1145/3341162.3344848]
  4. 4.Sadia Sharmin, Farzana Aktar, Amin Ahsan Ali, Muhammad Asif Hossain Khan, Mohammad Shoyaib. (2017). BFSp: A feature selection method for bug severity classification. , 750–754[10.1109/r10-htc.2017.8289066]
  5. 5.Monalisha Mahapatra, Santosh Kumar Majhi, Sunil Kumar Dhal. (2021). MRMR-SSA: a hybrid approach for optimal feature selection. Evolutionary Intelligence, 15(3), 2017–2036[10.1007/s12065-021-00608-8]
  6. 6.Puloma Roy, Sadia Sharmin, Amin Ahsan Ali, Mohammad Shoyaib. (2020). Discretization and Feature Selection Based on Bias Corrected Mutual Information Considering High-Order Dependencies. Lecture notes in computer science, 830–842[10.1007/978-3-030-47426-3_64]
  7. 7.Partha Ghosh, Sumit Biswas, Shivam Shakti, Santanu Phadikar. (2019). An Improved Intrusion Detection System to Preserve Security in Cloud Environment. International Journal of Information Security and Privacy, 14(1), 67–80[10.4018/ijisp.2020010105]
  8. 8.Md Nazmul Haque, Sadia Sharmin, Amin Ahsan Ali, Abu Ashfaqur Sajib, Mohammad Shoyaib. (2021). Use of relevancy and complementary information for discriminatory gene selection from high-dimensional gene expression data. PLoS ONE, 16(10), e0230164[10.1371/journal.pone.0230164]
  9. 9.Jun Qian, Yingchi Mao, Jianghong Tang, Longbao Wang. (2019). Feature Selection Method Based on Maximum Conditional and Joint Mutual Information. , 607–611[10.1109/icivc47709.2019.8981340]
  10. 10.Suravi Akhter, Sumon Ahmed, Ahmedul Kabir, Mohammad Shoyaib. (2022). A Relief Based Feature Subset Selection Method. Dhaka University Journal of Applied Science and Engineering, 6(2), 7–13[10.3329/dujase.v6i2.59214]
  11. 11.Md. Hasan Tarek, Md. Mumtahin Habib Ullah Mazumder, Sadia Sharmin, Md. Shariful Islam, Mohammad Shoyaib, Muhammad Mahbub Alam. (2022). RHC: Cluster based Feature Reduction for Network Intrusion Detections. , 15, 378–384[10.1109/ccnc49033.2022.9700680]
  12. 12.Sayeda Shamma Alia, Md. Nazmul Haque, Sadia Sharmin, Shah Mostafa Khaled, Mohammad Shoyaib. (2018). Bug Severity Classification Based on Class-Membership Information. , 97, 520–525[10.1109/iciev.2018.8641032]
  13. 13.Dong Xu, Xin Wang, Yulong Meng, Ziying Zhang. (2020). A Discretization Algorithm Based on Forest Optimization Network and Variable Precision Rough Set. Xibei Gongye Daxue Xuebao/Journal of Northwestern Polytechnical University, 38(2), 434–441[10.1051/jnwpu/20203820434]
  14. 14.Hongbin Dong, Xue Wang, Xingmei Wang, Jing Sun, Tao Li. (2019). A Feature Selection Method Based on Adaptive Differential Evolution. , 208–213[10.1109/icis46139.2019.8940171]
  15. 15.Jerzy W. Grzymala-Busse, Zdzislaw S. Hippe, Teresa Mroczek. (2019). Reduced Data Sets and Entropy-Based Discretization. Entropy, 21(11), 1051[10.3390/e21111051]
  16. 16.Ganga Gudi,, Dr. Mallamma V Reddy, Dr. Hanumanthappa M. (2023). Efficient Approach for Braille Conversion of Multilingual Text: Strategic Mapping Solutions. International Journal of Engineering Technology and Management Sciences, 7(4), 653–659[10.46647/ijetms.2023.v07i04.091]
  17. 17.Md Nazmul Haque, Sadia Sharmin, Amin Ahsan Ali, Abu Ashfaqur Sajib, Mohammad Shoyaib. (2020). Use of relevancy and complementary information for discriminatory gene selection from high-dimensional cancer data. bioRxiv (Cold Spring Harbor Laboratory)[10.1101/2020.02.25.964304]
← All publications