Journal Article2020Cited by 40

Human activity recognition from wearable sensor data using self-attention

Saif Mahmud, M Tonmoy, Kishor Kumar Bhaumik, AK Mahbubur Rahman, M Ashraful Amin, Mohammad Shoyaib, Muhammad Asif Hossain Khan, Amin Ahsan Ali

arXiv preprint arXiv:2003.09018

IOS Press, ISBN: 0922-6389

DOI: 10.3233/faia200236Open AccessURL

Abstract

Human Activity Recognition from body-worn sensor data poses an inherent challenge in capturing spatial and temporal dependencies of time-series signals. In this regard, the existing recurrent or convolutional or their hybrid models for activity recognition struggle to capture spatio-temporal context from the feature space of sensor reading sequence. To address this complex problem, we propose a self-attention based neural network model that foregoes recurrent architectures and utilizes different types of attention mechanisms to generate higher dimensional feature representation used for classification. We performed extensive experiments on four popular publicly available HAR datasets: PAMAP2, Opportunity, Skoda and USC-HAD. Our model achieve significant performance improvement over recent state-of-the-art models in both benchmark test subjects and Leave-one-subject-out evaluation. We also observe that the sensor attention maps produced by our model is able capture the importance of the modality and placement of the sensors in predicting the different activity classes.

Keywords

  • human activity recognition
  • self-attention
  • wearable sensors
  • Opportunity dataset
  • leave-one-subject-out evaluation
  • spatiotemporal dependencies
  • sensor placement
  • PAMAP2
  • attention maps

CCDS Authors

Saif Mahmud
Saif Mahmud

Saif Mahmud

  • Research Assistant, CCDS

PhD Student

Cornell University

Published at

PAKDD

Research Interests

Machine Learning

Kishor Kumar Bhaumik
Kishor Kumar Bhaumik

Kishor Kumar Bhaumik

PhD student

University of California, Riverside

Published at

WACV, Pattern Recognition

Research Interests

Machine Learning

Prof. M Ashraful Amin, PhD
M Ashraful Amin, PhD

Prof. M Ashraful Amin, PhD

  • Founder Director, CCDS
  • Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Published at

ACL, CHI, UbiComp, ECAI, WACV, PAKDD, IJCNN, ICPR, ICIP, HCII, CLEF, BLP, Scientific Reports, JMIR, IEEE Access, Smart Agricultural Technology, Frontiers in Computational Neuroscience, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

Amin Ahsan Ali, PhD
Amin Ahsan Ali, PhD

Amin Ahsan Ali, PhD

  • Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

ACL, CHI, UbiComp, ECCV, IJCAI, ECAI, WACV, IPSN, PAKDD, LREC, IJCNN, ICPR, ICIP, BEA, BLP, ICMLA, Scientific Reports, Scientific Data, Pattern Recognition, Information Sciences, Ad Hoc Networks, Drug and Alcohol Dependence, PLOS ONE

Research Interests

Artificial Intelligence, Machine Learning

Cited By

  1. 1.Bulat Khaertdinov, Esam Ghaleb, Stylianos Asteriadis. (2021). Contrastive Self-supervised Learning for Sensor-based Human Activity Recognition. , 1–8[10.1109/ijcb52358.2021.9484410]
  2. 2.Ye Zhang, Longguang Wang, Huiling Chen, Aosheng Tian, Shilin Zhou, Yulan Guo. (2022). IF-ConvTransformer. Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies, 6(2), 1–26[10.1145/3534584]
  3. 3.Razvan Brinzea, Bulat Khaertdinov, Stylianos Asteriadis. (2022). Contrastive Learning with Cross-Modal Knowledge Mining for Multimodal Human Activity Recognition. 2022 International Joint Conference on Neural Networks (IJCNN), 01–08[10.1109/ijcnn55064.2022.9892522]
  4. 4.Harish Haresamudram, Chi Ian Tang, Sungho Suh, Paul Lukowicz, Thomas Plötz. (2025). Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task. Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies, 9(2), 1–44[10.1145/3729467]
  5. 5.Xuan Li, Hengxin Chen, Shengdong He, Xinrun Chen, Shuang Dong, Ping Yan, Bin Fang. (2023). Action recognition based on multimode fusion for VR online platform. Virtual Reality, 27(3), 1797–1812[10.1007/s10055-023-00773-4]
  6. 6.Tonmoy, M Tanjid Hasan, Mahmud, Saif, Rahman, A K M Mahbubur, Amin, M Ashraful, Ali, Amin Ahsan. (2021). Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Recognition. arXiv (Cornell University)[10.1007/978-3-030-75768-7_28]
  7. 7.Feng Xu, Xuchen Gao, Weigang Wang. (2025). A human activity recognition model based on deep neural network integrating attention mechanism. Scientific Reports, 15(1), 23192[10.1038/s41598-025-98763-w]
  8. 8.Yuxin Zhang, Pari Delir Haghighi, Frada Burstein, Lina Yao, Flavia Cicuttini. (2021). On-Device Lumbar-Pelvic Movement Detection Using Dual-IMU: A DNN-Based Approach. IEEE Access, 9, 62241–62254[10.1109/access.2021.3074755]
  9. 9.Jiaoyi Wu, Hexiang Zhang, Hexiang Zhang, Enzan Xiao, Tianshuang Liang, Xiaolong Zou, Jiantong Sun, Chengliang Fan, Zutao Zhang. (2025). Driver abnormal behavior detection enabled self-powered magnetic suspension hybrid wristband and AI for smart transportation. Energy Conversion and Management, 326, 119485[10.1016/j.enconman.2025.119485]
  10. 10.Ayokunle Olalekan Ige, Mohd Halim Mohd Noor. (2022). A lightweight deep learning with feature weighting for activity recognition. Computational Intelligence, 39(2), 315–343[10.1111/coin.12565]
  11. 11.Emanuele Lattanzi, Lorenzo Calisti, Chiara Contoli. (2024). Are Transformers a Useful Tool for Tiny devices in Human Activity Recognition?. , 339–344[10.1145/3704137.3704171]
  12. 12.Yan Mao, Guoyin Zhang, Cuicui Ye. (2024). A Spatio-temporal Graph Transformer driven model for recognizing fine-grained data human activity. Alexandria Engineering Journal, 104, 31–45[10.1016/j.aej.2024.06.042]
  13. 13.Rebeen Ali Hamad, Wai Lok Woo, Bo Wei, Longzhi Yang. (2025). Self-Supervised Learning for Activity Recognition Based on Datasets With Imbalanced Classes. IEEE Transactions on Emerging Topics in Computational Intelligence, 9(5), 3441–3454[10.1109/tetci.2025.3526504]
  14. 14.Taoran Sheng, Manfred Huber. (2025). Reducing Label Dependency in Human Activity Recognition with Wearables: From Supervised Learning to Novel Weakly Self-Supervised Approaches. Sensors, 25(13), 4032[10.3390/s25134032]
  15. 15.Andrés Tello, Victoria Degeler. (2025). Contrasting Global and Local Representations for Human Activity Recognition using Graph Neural Networks. , 630–637[10.1145/3672608.3707743]
  16. 16.Md. Hasan Tarek, Md. Eusha Kadir, Mahir Mahbub, Pritom Saha Akash, Amin Ahsan Ali, Mohammad Shoyaib. (2020). Mutual Information based Feature Selection for Nurse Care Activity Recognition. , 1–6[10.1109/icievicivpr48672.2020.9306645]
  17. 17.Vanessa Camilleri, Reno Yuri Camilleri, Mark Fialovszky, Daniel Pace, Dylan Seychell, Matthew Montebello. (2025). Towards Explainable Multimodal Sensing for Swimming Analysis: Early Findings from the SWIM-360 Project. Sensors, 25(22), 7047[10.3390/s25227047]
  18. 18.Vitor Fortes Rey, Pedro Martelleto Bressane Rezende, Bo Zhou, Sungho Suh, Paul Lukowicz. (2025). COA-HAR: Exploring contrastive online test-time adaptation for wearable sensor-based human activity recognition using sensor data augmentation. Expert Systems with Applications, 297, 129288[10.1016/j.eswa.2025.129288]
  19. 19.Moumita Bhattacharya, Qingquan Sun, Jiang Lu, Yunfei Hou. (2025). Application of Transformer to Wearable Sensor-based Complex Human Activity Recognition. , 585–588[10.1109/icaibd64986.2025.11082055]
  20. 20.Mehdi Ejtehadi, Gloria Edumaba Graham, Cailin Ringstrom, Elisa Du, Robert Riener, Diego Paez-Granados. (2025). Tifex-Py: Time-Series Feature Extraction for Python in a Human Activity Recognition Scenario. , 2025, 1332–1339[10.1109/icorr66766.2025.11062978]
  21. 21.Su Hu, Junyu Zeng, Ximing Lin, Xin Zhong, Fangyi Liang, Peixi Chen, Xuemei Bai. (2025). Accident LLM: A multimodal large language model of accident description based on Inter-Frame Attention Mechanism. , 345–350[10.1109/iscait64916.2025.11010470]
  22. 22.Qian Huang, ZhongQi Chen, Chang Li, WeiWen Qian. (2025). Human Action Recognition in Multi-Level Convolutional Temporal Attention Network. , 1–5[10.1109/icassp49660.2025.10889140]
  23. 23.Yichao Diao, Hoi Leong Lee, Gang Jin. (2026). MST-HDQ: Multi-Scale temporal hash discriminative quantization for wearable sensor sequence recognition. Journal of King Saud University - Computer and Information Sciences, 38(7)[10.1007/s44443-026-01053-2]
  24. 24.Monu Nagar, Himani Sikarwar, Debasis Das. (2026). Enhancing Cross-User Generalization in Human Activity Recognition With Adaptive Noise Scheduling and Linear Attention. IEEE Sensors Journal, 26(14), 21626–21635[10.1109/jsen.2026.3694802]
  25. 25.Zhonghang Li, Fucheng Miao, Wei Zhang, Guan Gui. (2026). Wearable Sensor-Based Human Activity Recognition Using Deep Spiking Convolutional LSTM with Residual Attention. , 195–199[10.1109/ictc70246.2026.11650001]
  26. 26.Yaxiong Lei, Hyochan Cho, Fergus Buchanan, Shijing He, Xinya Gong, Yuheng Wang, Juan Ye. (2026). TinyGaze: Lightweight Gaze-Gesture Recognition on Commodity Mobile Devices. , 1–6[10.1145/3772363.3798475]
  27. 27.Francisco M. Calatrava-Nicolas, Vitor Fortes Rey, Paul Lukowicz, Todor Stoyanov, Oscar Martinez Mozos. (2026). Rotation Estimation Auxiliary Task for Wrist Inertial-Based Human Activity Recognition. , 1–6[10.1109/percomworkshops68308.2026.11585279]
  28. 28.Priyanka Kaushik, Jigar Soni, Bhawna, Neeraj Varshney, Anshika Tyagi, Krishan Kumar Garg. (2026). Design and Implementation of an Artificial Intelligence—Based System for Stock Market Prediction and Investment Insights. , 1–5[10.1109/iatmsi68868.2026.11465588]
  29. 29.Oktay Ozan Güner, Sergio Consoli, Vicenç Gómez, Mario Ceresa. (2026). Differentially Private Fine-Tuning of Self-Supervised Learning Models for Human Activity Recognition on Wearables. Communications in computer and information science, 3–20[10.1007/978-3-032-16708-8_1]
  30. 30.Taoran Sheng, Manfred Huber. (2025). Label-Efficient Human Activity Recognition from Wearables via Self-Supervised Representation Learning. , 316–323[10.1109/icmla66185.2025.00049]
  31. 31.E. Riveros, D. Vega-Oliveros, A. Soriano-Vargas, A. Rocha. (2025). Towards Intelligent Dietary Monitoring: Deep Learning Techniques for Alimentary Activity Detection. , 604–608[10.1109/iscmi67495.2025.11357866]
  32. 32.Anjani Kumar Reddy Challa, Vamshi Krishna Raavi, Harshavardan Bandari, Dhanvanth Kumar Gude, Siddharth Kumar. (2025). Human Activity Recognition Reimagined: The CGRNHAR Self-Supervised Approach. , 359–364[10.1109/aisummit66170.2025.11411187]
  33. 33.Aadhira Suleim A R, Anlin Sahaya Infant Tinu M. (2025). Smart Wearable Sensor System for Real-Time Biomechanical Analysis with AI Integration. , 1–6[10.1109/icbmesh66209.2025.11182236]
  34. 34.Meenambika Chandirakumar, Thanushanth Kanagarajah, Nithursika Kalanantharasan, Sandareka Wickramanayake, Dulani Meedeniya. (2025). Human Activity Recognition Using Spatio-Temporal Dual Attention with Cross-Sensor Attention. , 1–8[10.1109/ijcnn64981.2025.11228134]
  35. 35.Jinghua Yan, Wenkai Zhou, Wei Zhang, Yuchao Lu, Lin Huang. (2025). MHCAGE:Multi-Head Attention-based Combined Accelerometer and Gyroscope Embedding Network. , 631–636[10.1109/aemcse65292.2025.11042562]
  36. 36.Jiangtao Fan, Anish Jindal, Amir Atapour-Abarghouei. (2025). Multi-scale Efficient Spatial Attention on Human Activity Recognition Using Wearable Sensors. Communications in computer and information science, 150–164[10.1007/978-981-96-6963-9_11]
  37. 37.Haoning Wu, Shaowu Wu, Zhicheng Zhang, Hao Zhou, Xiaoping Wu, Xiaoguang Niu, Duo Li. (2025). MPAM: Dual-Transformer for Millimeter-Wave Sensing Based Multi-person Activity Monitoring System. Lecture notes in computer science, 320–331[10.1007/978-981-96-8728-2_26]
  38. 38.Jiancong Ye, Mengxuan Wang, Junpei Zhong, Hongjie Jiang. (2024). A review on devices and learning techniques in domestic intelligent environment. Journal of Ambient Intelligence and Humanized Computing, 15(4), 2361–2380[10.1007/s12652-024-04759-1]
  39. 39.Bulat Khaertdinov. (2024). Feature representation learning for human activity recognition. [10.26481/dis.20240618bk]
  40. 40.Linh Trinh, Bach Ha. (2022). An incorporation of deep temporal convolutional networks with hidden markov models post-processing for sensor-based human activity recognition. , 96–102[10.1145/3568562.3568610]
← All publications