Conference Paper2019Cited by 4

Introducing confidence as a weight in random forest

Pritom Saha Akash, Md Eusha Kadir, Amin Ahsan Ali, Md Nurul Ahad Tawhid, Mohammad Shoyaib

2019 International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST)

IEEE, pp. 611–616

DOI: 10.1109/icrest.2019.8644396URL

Abstract

Random forest is one of the most popular supervised learning methods which is a collection of multiple decision trees. It is computationally fast, easy to use and gives reasonable performances over diversified applications. In a random forest, outcomes from multiple trees are aggregated to make the final decision where all trees get the same importance. However, it is natural that the performance of all trees is not equal and thus calculating the majority vote from these trees may not always give the correct decision. To improve this situation, we propose Confidence weighted Random Forest (CwRF) which uses a confidence factor for each leaf node in every decision tree and use that confidence to take weighted majority vote for making the final decision. A rigorous experiment over twenty-five datasets illustrates that the proposed CwRF outperforms three other state-of-the-art methods and provides significantly better performance over nineteen datasets out of twenty-five datasets at best.

Keywords

  • Random forest
  • Decision tree
  • weighted majority vote
  • confidence weighting
  • Ensemble learning
  • classifier aggregation

CCDS Authors

Amin Ahsan Ali, PhD
Amin Ahsan Ali, PhD

Amin Ahsan Ali, PhD

  • Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

ACL, CHI, UbiComp, ECCV, IJCAI, ECAI, WACV, IPSN, PAKDD, LREC, IJCNN, ICPR, ICIP, BEA, BLP, ICMLA, Scientific Reports, Scientific Data, Pattern Recognition, Information Sciences, Ad Hoc Networks, Drug and Alcohol Dependence, PLOS ONE

Research Interests

Artificial Intelligence, Machine Learning

References

  1. 1.Leo Breiman. (2001). Random Forests. Machine Learning, 45(1), 5–32[10.1023/a:1010933404324]
  2. 2.Vladimir N. Vapnik. (1995). The Nature of Statistical Learning Theory. [10.1007/978-1-4757-2440-0]
  3. 3.Trevor Hastie, Jerome Friedman, Robert Tibshirani. (2001). The Elements of Statistical Learning. Springer series in statistics[10.1007/978-0-387-21606-5]
  4. 4.Pierre Geurts, Damien Ernst, Louis Wehenkel. (2006). Extremely randomized trees. Machine Learning, 63(1), 3–42[10.1007/s10994-006-6226-1]
  5. 5.S.R. Safavian, D. Landgrebe. (1991). A survey of decision tree classifier methodology. IEEE Transactions on Systems Man and Cybernetics, 21(3), 660–674[10.1109/21.97458]
  6. 6.Thomas G. Dietterich. (1998). Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation, 10(7), 1895–1923[10.1162/089976698300017197]
  7. 7.Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, Andrew Blake. (2011). Real-time human pose recognition in parts from single depth images. , 1297–1304[10.1109/cvpr.2011.5995316]
  8. 8.Zhi-Hua Zhou, Jianxin Wu, Wei Tang. (2002). Ensembling neural networks: Many could be better than all. Artificial Intelligence, 137(1-2), 239–263[10.1016/s0004-3702(02)00190-x]
  9. 9.Shaoqing Ren, Xudong Cao, Yichen Wei, Jian Sun. (2014). Face Alignment at 3000 FPS via Regressing Local Binary Features. , 1685–1692[10.1109/cvpr.2014.218]
  10. 10.A. Verikas, A. Gelzinis, M. Bacauskiene. (2010). Mining data with random forests: A survey and results of new tests. Pattern Recognition, 44(2), 330–349[10.1016/j.patcog.2010.08.011]
  11. 11.V. Lepetit, P. Fua. (2006). Keypoint recognition using randomized trees. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(9), 1465–1479[10.1109/tpami.2006.188]
  12. 12.Bjoern H. Menze, B. Michael Kelm, Daniel N. Splitthoff, Ullrich Koethe, Fred A. Hamprecht. (2011). On Oblique Random Forests. Lecture notes in computer science, 453–469[10.1007/978-3-642-23783-6_29]
  13. 13.Hyunjoong Kim, Hyunjoong Kim, Hyeuk Kim, Hyeuk Kim, Hojin Moon, Hongshik Ahn. (2011). A weight-adjusted voting algorithm for ensembles of classifiers. Journal of the Korean Statistical Society, 40(4), 437–449[10.1016/j.jkss.2011.03.002]
  14. 14.Stacey J. Winham, Robert R. Freimuth, Joanna M. Biernacka. (2013). A weighted random forests approach to improve predictive performance. Statistical Analysis and Data Mining The ASA Data Science Journal, 6(6), 496–505[10.1002/sam.11196]
  15. 15.Shaoqing Ren, Xudong Cao, Yichen Wei, Jian Sun. (2015). Global refinement of random forest. , 723–730[10.1109/cvpr.2015.7298672]
  16. 16.Jesús Maudes, Juan J. Rodríguez, César García-Osorio, Nicolás García-Pedrajas. (2010). Random feature weights for decision tree ensemble construction. Information Fusion, 13(1), 20–30[10.1016/j.inffus.2010.11.004]
  17. 17.Nan Li, Zhi-Hua Zhou. (2009). Selective Ensemble under Regularization Framework. Lecture notes in computer science, 293–303[10.1007/978-3-642-02326-2_30]
  18. 18.Qiu, Qiang, Sapiro, Guillermo. (2013). Learning Transformations for Classification Forests. arXiv (Cornell University)[10.48550/arxiv.1312.5604]

Cited By

  1. 1.Sebastian Buschjäger, Katharina Morik. (2023). Joint leaf-refinement and ensemble pruning through $$L_1$$ regularization. Data Mining and Knowledge Discovery, 37(3), 1230–1261[10.1007/s10618-023-00921-z]
  2. 2.Vikas Jain, Ashish Phophalia. (2020). M-ary Random Forest - A new multidimensional partitioning approach to Random Forest. Multimedia Tools and Applications, 80(28-29), 35217–35238[10.1007/s11042-020-10047-9]
  3. 3.Vikas Jain, Jaya Sharma, Kriti Singhal, Ashish Phophalia. (2019). Exponentially Weighted Random Forest. Lecture notes in computer science, 170–178[10.1007/978-3-030-34869-4_19]
  4. 4.Sascha Meyen, Frieder Göppert, Carina Schrenk, Ulrike von Luxburg, Volker H. Franz. (2025). Information-Theoretic Measures of Metacognition: Bounds and Relation to Group Performance. Open Mind, 9, 1728–1762[10.1162/opmi.a.40]
← All publications