Conference Paper2019Cited by 26

Gru-based attention mechanism for human activity recognition

Md Nazmul Haque, M Tanjid Hasan Tonmoy, Saif Mahmud, Amin Ahsan Ali, Muhammad Asif Hossain Khan, Mohammad Shoyaib

2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT)

IEEE, pp. 1–6

DOI: 10.1109/icasert.2019.8934659URL

Abstract

Sensor data based Human Activity Recognition (HAR) has gained interest due to its application in practical field. With increasing number of approaches incorporating feature learning of sequential time-series sensor data, in particular the deep learning based ones has performed reasonably in uniform labeled data distribution scenario. However, most of these methods do not capture properly the temporal context of time-steps in sequential time-series data. Moreover, the situation becomes worse for imbalanced class distribution which is a usual case for HAR using body-worn sensor devices. To solve this issues, we have integrated hierarchical attention mechanism with recurrent units of neural network in order to obtain temporal context within the time-steps of data sequence. The introduced model in this paper has achieved better performance with respect to the well-defined evaluation metrics in both uniform and imbalanced class distribution than the existing state-of-the-art deep learning based model.

Keywords

  • human activity recognition
  • class imbalance
  • wearable sensors
  • hierarchical attention mechanism
  • recurrent neural networks
  • time-series sensor data
  • temporal context
  • GRU

CCDS Authors

Saif Mahmud
Saif Mahmud

Saif Mahmud

  • Research Assistant, CCDS

PhD Student

Cornell University

Published at

PAKDD

Research Interests

Machine Learning

Amin Ahsan Ali, PhD
Amin Ahsan Ali, PhD

Amin Ahsan Ali, PhD

  • Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

ACL, CHI, UbiComp, ECCV, IJCAI, ECAI, WACV, IPSN, PAKDD, LREC, IJCNN, ICPR, ICIP, BEA, BLP, ICMLA, Scientific Reports, Scientific Data, Pattern Recognition, Information Sciences, Ad Hoc Networks, Drug and Alcohol Dependence, PLOS ONE

Research Interests

Artificial Intelligence, Machine Learning

References

  1. 1.Sepp Hochreiter, Jürgen Schmidhuber. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780[10.1162/neco.1997.9.8.1735]
  2. 2.Kingma, Diederik P., Ba, Jimmy. (2014). Adam: A Method for Stochastic Optimization. UvA-DARE (University of Amsterdam)[10.48550/arxiv.1412.6980]
  3. 3.Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90[10.1145/3065386]
  4. 4.Mandi, Jayanta, Canoy, Rocsildes, Bucarey, Víctor, Guns, Tias. (2021). DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)[10.4230/lipics.cp.2021.42]
  5. 5.Laurens van der Maaten, Geoffrey E. Hinton. (2008). Visualizing Data using t-SNE. Journal of Machine Learning Research, 9(86), 2579–2605
  6. 6.Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov. (2014). Dropout: a simple way to prevent neural networks from overfitting. , 15(1), 1929–1958
  7. 7.Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, Yoshua Bengio. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. , 1724–1734[10.3115/v1/d14-1179]
  8. 8.Y. Bengio. (2009). Learning Deep Architectures for AI. Foundations and Trends® in Machine Learning, 2(1), 1–127[10.1561/2200000006]
  9. 9.Y. Bengio. (2009). Learning Deep Architectures for AI. now publishers, Inc. eBooks[10.1561/9781601982957]
  10. 10.Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, Eduard Hovy. (2016). Hierarchical Attention Networks for Document Classification. , 1480–1489[10.18653/v1/n16-1174]
  11. 11.Ling Bao, Stephen S. Intille. (2004). Activity Recognition from User-Annotated Acceleration Data. Lecture notes in computer science, 1–17[10.1007/978-3-540-24646-6_1]
  12. 12.Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz. (2013). A public domain dataset for human activity recognition using smartphones. RECERCAT (Consorci de Serveis Universitaris de Catalunya), 437–442[link]
  13. 13.D.W.F. Van Krevelen, R. Poelman. (2010). A Survey of Augmented Reality Technologies, Applications and Limitations. International Journal of Virtual Reality, 9(2), 1–20[10.20870/ijvr.2010.9.2.2767]
  14. 14.Charissa Ann Ronao, Sung-Bae Cho. (2016). Human activity recognition with smartphone sensors using deep learning neural networks. Expert Systems with Applications, 59, 235–244[10.1016/j.eswa.2016.04.032]
  15. 15.Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge L. Reyes-Ortiz. (2012). Human Activity Recognition on Smartphones Using a Multiclass Hardware-Friendly Support Vector Machine. Lecture notes in computer science, 216–223[10.1007/978-3-642-35395-6_30]
  16. 16.Jian Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li, Shonali Krishnaswamy. (2015). Deep convolutional neural networks on multichannel time series for human activity recognition. , 3995–4001
  17. 17.Moez Baccouche, Franck Mamalet, Christian Wolf, Christophe Garcia, Atilla Baskurt. (2011). Sequential Deep Learning for Human Action Recognition. Lecture notes in computer science, 29–39[10.1007/978-3-642-25446-8_4]
  18. 18.Li Deng. (2014). A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPA Transactions on Signal and Information Processing, 3(1)[10.1017/atsip.2013.9]
  19. 19.Charles X. Ling, Jin Huang, Harry Zhang. (2003). AUC: A Better Measure than Accuracy in Comparing Learning Algorithms. Lecture notes in computer science, 329–341[10.1007/3-540-44886-1_25]
  20. 20.Huan Song, Deepta Rajan, Jayaraman Thiagarajan, Andreas Spanias. (2018). Attend and Diagnose: Clinical Time Series Analysis Using Attention Models. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1)[10.1609/aaai.v32i1.11635]
  21. 21.Wanmin Wu, Sanjoy Dasgupta, Ernesto E Ramirez, Carlyn Peterson, Gregory J Norman. (2012). Classification Accuracies of Physical Activities Using Smartphone Motion Sensors. Journal of Medical Internet Research, 14(5), e130[10.2196/jmir.2208]
  22. 22.Y. Bengio, P. Frasconi, P. Simard. (2002). The problem of learning long-term dependencies in recurrent networks. , 1183–1188[10.1109/icnn.1993.298725]
  23. 23.Damla Arifoglu, Abdelhamid Bouchachia. (2017). Activity Recognition and Abnormal Behaviour Detection with Recurrent Neural Networks. Procedia Computer Science, 110, 86–93[10.1016/j.procs.2017.06.121]
  24. 24.Sadia Sharmin, Mohammad Shoyaib, Amin Ahsan Ali, Muhammad Asif Hossain Khan, Oksam Chae. (2019). Simultaneous feature selection and discretization based on mutual information. Pattern Recognition, 91, 162–174[10.1016/j.patcog.2019.02.016]
  25. 25.Nguyen Xuan Vinh, Shuo Zhou, Jeffrey Chan, James Bailey. (2015). Can high-order dependencies improve mutual information based feature selection?. Pattern Recognition, 53, 46–58[10.1016/j.patcog.2015.11.007]
  26. 26.Ming Zeng, Tong Yu, Xiao Wang, Le T. Nguyen, Ole J. Mengshoel, Ian Lane. (2017). Semi-supervised convolutional neural networks for human activity recognition. , 522–529[10.1109/bigdata.2017.8257967]
  27. 27.Stefan Duffner, Samuel Berlemont, Gregoire Lefebvre, Christophe Garcia. (2014). 3D gesture classification with convolutional neural networks. , 5432–5436[10.1109/icassp.2014.6854641]
  28. 28.Yuanliang Meng, Anna Rumshisky. (2018). Context-Aware Neural Model for Temporal Information Extraction. , 527–536[10.18653/v1/p18-1049]
  29. 29.Christian Vollmer, Horst-Michael Gross, Julian P. Eggert. (2013). Learning Features for Activity Recognition with Shift-Invariant Sparse Coding. Lecture notes in computer science, 367–374[10.1007/978-3-642-40728-4_46]
  30. 30.Ali Mahmood Khan. (2013). Recognizing Physical Activities Using Wii Remote. International Journal of Information and Education Technology, 60–62[10.7763/ijiet.2013.v3.234]
  31. 31.F. Montalto, C. Guerra, V. Bianchi, I. De Munari, P. Ciampolini. (2015). MuSA: Wearable Multi Sensor Assistant for Human Activity Recognition and Indoor Localization. Biosystems & biorobotics, 81–92[10.1007/978-3-319-18374-9_8]
  32. 32.Bahdanau, Dzmitry, Cho, Kyunghyun, Bengio, Yoshua. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv (Cornell University)[10.48550/arxiv.1409.0473]
  33. 33.Chung, Junyoung, Gulcehre, Caglar, Cho, KyungHyun, Bengio, Yoshua. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv (Cornell University)[10.48550/arxiv.1412.3555]
  34. 34.Zeiler, Matthew D.. (2012). ADADELTA: An Adaptive Learning Rate Method. arXiv (Cornell University)[10.48550/arxiv.1212.5701]
  35. 35.Chorowski, Jan, Bahdanau, Dzmitry, Serdyuk, Dmitriy, Cho, Kyunghyun, Bengio, Yoshua. (2015). Attention-Based Models for Speech Recognition. arXiv (Cornell University)[10.48550/arxiv.1506.07503]
  36. 36.Yin, Wenpeng, Kann, Katharina, Yu, Mo, Schütze, Hinrich. (2017). Comparative Study of CNN and RNN for Natural Language Processing. arXiv (Cornell University)[10.48550/arxiv.1702.01923]
  37. 37.Raffel, Colin, Ellis, Daniel P. W.. (2015). Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems. arXiv (Cornell University)[10.48550/arxiv.1512.08756]
  38. 38.Li Deng. (2014). A tutorial survey of architectures, algorithms, and applications for deep learning – ERRATUM. APSIPA Transactions on Signal and Information Processing, 3(1)[10.1017/atsip.2014.4]

Cited By

  1. 1.Sakorn Mekruksavanich, Anuchit Jitpattanakul. (2022). RNN-based deep learning for physical activity recognition using smartwatch sensors: A case study of simple and complex activity recognition. Mathematical Biosciences & Engineering, 19(6), 5671–5698[10.3934/mbe.2022265]
  2. 2.Mahmud, Saif, Tonmoy, M Tanjid Hasan, Bhaumik, Kishor Kumar, Rahman, A K M Mahbubur, Amin, M Ashraful, Shoyaib, Mohammad, Khan, Muhammad Asif Hossain, Ali, Amin Ahsan. (2020). Human Activity Recognition from Wearable Sensor Data Using Self-Attention. arXiv (Cornell University)[10.48550/arxiv.2003.09018]
  3. 3.Xiaojie Sun, Hongji Xu, Zheng Dong, Leixin Shi, Qiang Liu, Juan Li, Tiankuo Li, Shidi Fan, Yuhao Wang. (2022). CapsGaNet: Deep Neural Network Based on Capsule and GRU for Human Activity Recognition. IEEE Systems Journal, 16(4), 5845–5855[10.1109/jsyst.2022.3153503]
  4. 4.Ruohong Huan, Chengxi Jiang, Luoqi Ge, Jia Shu, Ziwei Zhan, Peng Chen, Kaikai Chi, Ronghua Liang. (2021). Human Complex Activity Recognition With Sensor Data Using Multiple Features. IEEE Sensors Journal, 22(1), 757–775[10.1109/jsen.2021.3130913]
  5. 5.Md. Nazmul Haque, Mahir Mahbub, Md. Hasan Tarek, Lutfun Nahar Lota, Amin Ahsan Ali. (2019). Nurse care activity recognition. , 719–723[10.1145/3341162.3344848]
  6. 6.Xun Wang, Jie Shang. (2023). Human Activity Recognition Based on Two-Channel Residual–GRU–ECA Module with Two Types of Sensors. Electronics, 12(7), 1622[10.3390/electronics12071622]
  7. 7.Md. Khaliluzzaman, Md. Furquan, Mohammod Sazid Zaman Khan, Md. Jiabul Hoque. (2024). STA‐HAR: A Spatiotemporal Attention‐Based Framework for Human Activity Recognition. Applied Computational Intelligence and Soft Computing, 2024(1)[10.1155/2024/1832298]
  8. 8.Ruohong Huan, Ziwei Zhan, Luoqi Ge, Kaikai Chi, Peng Chen, Ronghua Liang. (2021). A hybrid CNN and BLSTM network for human complex activity recognition with multi-feature fusion. Multimedia Tools and Applications, 80(30), 36159–36182[10.1007/s11042-021-11363-4]
  9. 9.Taima Rahman Mim, Maliha Amatullah, Sadia Afreen, Mohammad Abu Yousuf, Shahadat Uddin, Salem A. Alyami, Khondokar Fida Hasan, Mohammad Ali Moni. (2022). Gru-Inc: An Inception-Attention Based Approach Using Gru for Human Activity Recognition. SSRN Electronic Journal[10.2139/ssrn.4167818]
  10. 10.Qian Huang, Weiliang Xie, Chang Li, Yanfang Wang, Yanwei Liu. (2023). Human Action Recognition Based on Hierarchical Multi-Scale Adaptive Conv-Long Short-Term Memory Network. Applied Sciences, 13(19), 10560[10.3390/app131910560]
  11. 11.Kyongseok Jang, Chao Sun, Junhao Zhou, Yongbin Seo, Youngok Kim, Seyeong Choi. (2025). A Study on the Lightweight and Fast Response GRU Techniques for Indoor Continuous Motion Recognition Based on Wi-Fi CSI. IEEE Access, 13, 36135–36150[10.1109/access.2025.3544627]
  12. 12.Ohoud Nafea, Wadood Abdul, Ghulam Muhammad. (2024). Incorporating Attention Mechanism Into CNN-BiGRU Classifier for HAR. IEEE Access, 12, 160205–160218[10.1109/access.2024.3487860]
  13. 13.Md. Sadman Siraj, Omar Shahid, Md Atiqur Rahman Ahad. (2020). Cooking Activity Recognition with Varying Sampling Rates Using Deep Convolutional GRU Framework. Smart innovation, systems and technologies, 115–126[10.1007/978-981-15-8269-1_10]
  14. 14.Aniruddh G. Fataniya, Hardik P. Modi. (2022). Comprehensive Analysis of Deep Learning-based Human Activity Recognition approaches based on Accuracy. International Journal of Computing and Digital Systems, 12(4), 1097–1118[10.12785/ijcds/120188]
  15. 15.Qingtian Zeng, Shuai Guo, Rui Cao, Ziqi Zhao, Hua Duan. (2023). Legal Transition Sequence Recognition of a Bounded Petri Net Using a Gate Recurrent Unit. IEEE Transactions on Big Data, 10(1), 66–76[10.1109/tbdata.2023.3319252]
  16. 16.Zulfiqar Ali, Israr Ur Rehman, Muhammad Abdul Basit Ur Rahim, Jonathan Witkowski. (2026). Enhancing Anomaly Detection in Software Logs with Bi-LSTM and Attention Mechanisms. Communications in computer and information science, 78–89[10.1007/978-3-032-08977-9_7]
  17. 17.Guoqiang Chen, Yong Li. (2023). Human Activity Recognition Based on Extraction of Multi-feature Using Inertial Sensors. , 519–524[10.1109/icftic59930.2023.10456356]
  18. 18.Md Readion Islam Razon, Md Amir Hamja, Md. Tanjim, Md Nafish Fuad Tushar, Md. Ziaul Hassan, Mahmudul Hasan. (2026). Asteroids, algorithms, and explainability: a unified framework for efficient classification of near-Earth objects. Experimental Astronomy, 62(1)[10.1007/s10686-026-10071-x]
  19. 19.(2025). Deep Learning-Based Recognition and Classification of Technical Errors in Squat Movements. , 681–686[10.1109/rivf68649.2025.11365151]
  20. 20.Nazish Ashfaq, Ibtsam Sadiq, Muhammad Junaid, Muhammad Hassan Khan, Muhammad Shahid Farid. (2024). Optimizing Human Activity Recognition with Ensemble Deep Learning on Wearable Sensor Data. International Journal of Innovations in Science and Technology, 1751[10.33411/ijist/1069]
  21. 21.Sayantani De, Ranjita Das, Ashish Singh Patel. (2024). Bengali Image Caption Generation using Attention Mechanism. , 1–6[10.1109/cvmi61877.2024.10781602]
  22. 22.Sonia Abraham, Rekha K. James. (2023). Significance of handcrafted features in human activity recognition with attention-based RNN models. International journal of electrical and computer engineering systems, 14(10), 1151–1163[10.32985/ijeces.14.10.8]
  23. 23.Weiliang Xie, Qian Huang, Chang Li, Yanfang Wang, Yanwei Liu. (2023). Hierarchical Multi-Scale Adaptive Conv-LSTM Network for Human Action Recognition Based on Wearable Sensors. , 1–8[10.1145/3595916.3626425]
  24. 24.Sonia Abraham, Rekha K. James. (2022). Smartphone Based Human Activity Recognition Using RNN Variants. 2022 IEEE 19th India Council International Conference (INDICON), 21.6, 1–5[10.1109/indicon56171.2022.10040026]
  25. 25.Aryan Karn, Dharm Raj Maurya. (2021). AN ADVANCED APPROACH TO RECOGNIZE HUMAN ACTIVITIES VIA DEEP LEARNING. International Journal of Engineering Applied Sciences and Technology, 6(1)[10.33564/ijeast.2021.v06i01.036]
  26. 26.(2019). ICASERT 2019 Table of Contents. 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), i–xx[10.1109/icasert.2019.8934687]
← All publications