Hierarchical self attention based autoencoder for open-set human activity recognition

M Tanjid Hasan Tonmoy; Saif Mahmud; AKM Mahbubur Rahman; M Ashraful Amin; Amin Ahsan Ali

doi:10.1007/978-3-030-75768-7_28

Conference Paper2021Cited by 2

Hierarchical self attention based autoencoder for open-set human activity recognition

M Tanjid Hasan Tonmoy, Saif Mahmud, AKM Mahbubur Rahman, M Ashraful Amin, Amin Ahsan Ali

Pacific-Asia Conference on Knowledge Discovery and Data Mining

Springer International Publishing, pp. 351–363, ISBN: 9783030757670

DOI: 10.1007/978-3-030-75768-7_28 Open Access URL

Keywords

Activity recognitionComputer scienceRobustness (evolution)AutoencoderArtificial intelligencePattern recognition (psychology)Wearable computerSet (abstract data type)EncoderMachine learning

CCDS Authors

Saif Mahmud

Research Assistant, CCDS

PhD Student

Cornell University

Research Interests

Machine Learning

AKM Mahbubur Rahman, PhD

Director, DS & IPT Wings

Associate Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Prof. M Ashraful Amin, PhD

Founder Director, CCDS
Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

Amin Ahsan Ali, PhD

Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Research Interests

Artificial Intelligence, Machine Learning

References

1.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin. (2025). Attention Is All You Need. , 30, 5998–6008[10.65215/2q58a426]
2.Francisco Ordóñez, Daniel Roggen. (2016). Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors, 16(1), 115[10.3390/s16010115]
3.Attila Reiss, Didier Stricker. (2012). Introducing a New Benchmarked Dataset for Activity Monitoring. , 108–109[10.1109/iswc.2012.13]
4.Jubai An. (2015). Variational autoencoder based anomaly detection using reconstruction probability. , 2, 1
5.Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Lukowicz, David Bannach, Gerald Pirkl, Alois Ferscha, Jakob Doppler, Clemens Holzmann, Marc Kurz, Gerald Holl, Ricardo Chavarriaga, Hesam Sagha, Hamidreza Bayati, Marco Creatura, José del R. Millán. (2010). Collecting complex activity datasets in highly rich networked sensor environments. , 233–240[10.1109/inss.2010.5573462]
6.Marc Bächlin, Meir Plotnik, Daniel Roggen, Inbal Maidan, Jeffrey M. Hausdorff, Nir Giladi, Gerhard Tröster. (2009). Wearable Assistant for Parkinson’s Disease Patients With the Freezing of Gait Symptom. IEEE Transactions on Information Technology in Biomedicine, 14(2), 436–446[10.1109/titb.2009.2036165]
7.Mi Zhang, Alexander A. Sawchuk. (2012). USC-HAD. , 1036–1043[10.1145/2370216.2370438]
8.Yu Guan, Thomas Plötz. (2017). Ensembles of Deep LSTM Learners for Activity Recognition using Wearables. Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies, 1(2), 1–28[10.1145/3090076]
9.Thomas Stiefmeier, Daniel Roggen, Georg Ogris, Paul Lukowicz, Gerhard Tr. (2008). Wearable Activity Tracking in Car Manufacturing. IEEE Pervasive Computing, 7(2), 42–50[10.1109/mprv.2008.40]
10.Haojie Ma, Wenzhong Li, Xiao Zhang, Songcheng Gao, Sanglu Lu. (2019). AttnSense: Multi-level Attention Mechanism For Multimodal Human Activity Recognition. , 3109–3115[10.24963/ijcai.2019/431]
11.Ming Zeng, Haoxiang Gao, Tong Yu, Ole J. Mengshoel, Helge Langseth, Ian Lane, Xiaobing Liu. (2018). Understanding and improving recurrent networks for human activity recognition by continuous attention. , 56–63[10.1145/3267242.3267286]
12.Liangying Peng, Ling Chen, Zhenan Ye, Yi Zhang. (2018). AROMA. Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies, 2(2), 1–16[10.1145/3214277]
13.Lili Meng, Bo Zhao, Bo Chang, Gao Huang, Wei Sun, Frederick Tung, Leonid Sigal. (2019). Interpretable Spatio-Temporal Attention for Video Action Recognition. , 1513–1522[10.1109/iccvw.2019.00189]
14.Yegang Du, Yuto Lim, Yasuo Tan. (2019). A Novel Human Activity Recognition and Prediction in Smart Home Based on Interaction. Sensors, 19(20), 4474[10.3390/s19204474]
15.Harish Haresamudram, David V. Anderson, Thomas Plötz. (2019). On the role of features in human activity recognition. , 78–88[10.1145/3341163.3347727]
16.Shang Gao, John X. Qiu, Mohammed Alawad, Jacob Hinkle, Noah Schaefferkoetter, Hong‐Jun Yoon, Blair Christian, Paul Fearn, Lynne Penberthy, Xiao‐Cheng Wu, Linda Coyle, Georgia D. Tourassi, Arvind Ramanathan. (2019). Classifying cancer pathology reports with hierarchical self-attention networks. Artificial Intelligence in Medicine, 101, 101726[10.1016/j.artmed.2019.101726]
17.Hangwei Qian, Sinno Jialin Pan, Bingshui Da, Chunyan Miao. (2019). A Novel Distribution-Embedded Neural Network for Sensor-Based Activity Recognition. , 5614–5620[10.24963/ijcai.2019/779]
18.Saif Mahmud, M Tanjid Hasan Tonmoy, Kumar Bhaumik Kishor, A K M Mahbubur Rahman, M. Ashraful Amin, Mohammad Shoyaib, Asif Hossain Khan Muhammad, Ahsan Ali Amin. (2020). Human Activity Recognition from Wearable Sensor Data Using Self-Attention. Frontiers in artificial intelligence and applications, 1332–1339[10.3233/faia200236]
19.Shuochao Yao, Yiran Zhao, Huajie Shao, Dongxin Liu, Shengzhong Liu, Yifan Hao, Ailing Piao, Shaohan Hu, Lü Su, Tarek Abdelzaher. (2019). SADeepSense: Self-Attention Deep Learning Framework for Heterogeneous On-Device Sensors in Internet of Things Applications. , 1243–1251[10.1109/infocom.2019.8737500]
20.Fadi Al Machot, Mohammed R. Elkobaisi, Kyandoghere Kyamakya. (2020). Zero-Shot Human Activity Recognition Using Non-Visual Sensors. Sensors, 20(3), 825[10.3390/s20030825]
21.Hyeokhyen Kwon, Gregory D. Abowd, Thomas Plötz. (2018). Adding structural characteristics to distribution-based accelerometer representations for activity recognition using wearables. , 72–75[10.1145/3267242.3267258]
22.Vishvak Murahari, Thomas Ploetz. (2018). On attention models for human activity recognition. , 100–103[10.1145/3267242.3267287]
23.Weihao Cheng, Sarah Erfani, Rui Zhang, Kotagiri Ramamohanarao. (2018). Predicting Complex Activities from Ongoing Multivariate Time Series. , 3322–3328[10.24963/ijcai.2018/461]
24.Zengwei Zheng, Lifei Shi, Chi Wang, Lin Sun, Gang Pan. (2019). LSTM with Uniqueness Attention for Human Activity Recognition. Lecture notes in computer science, 498–509[10.1007/978-3-030-30508-6_40]
25.Kandan Ramakrishnan, Mathew Monfort, Barry McNamara, Alex Lascelles, Dan Gutfreund, Rogério Feris, Aude Oliva. (2019). Identifying Interpretable Action Concepts in Deep Networks. Computer Vision and Pattern Recognition, 12–15

Cited By

1.Manuel Gil-Martín, Rubén San-Segundo, Fernando Fernández-Martínez, Javier Ferreiros-López. (2021). Time Analysis in Human Activity Recognition. Neural Processing Letters, 53(6), 4507–4525[10.1007/s11063-021-10611-w]
2.Marcel Dix, Jan Jens Koltermann, Sebastian Mieck, Heiko Petersen, Sebastian Taege, Gahana Anjanappa. (2024). Using Siamese Neural Networks for the Open Set Recognition of Anomalies Detected in Industrial Time Series Data. , 1–8[10.1109/etfa61755.2024.10710747]

← All publications

CCDS Authors

Saif Mahmud

Research Assistant, CCDS

PhD Student

Cornell University

Research Interests

Machine Learning

AKM Mahbubur Rahman, PhD

Director, DS & IPT Wings

Associate Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Prof. M Ashraful Amin, PhD

Founder Director, CCDS
Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

Amin Ahsan Ali, PhD

Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Research Interests

Artificial Intelligence, Machine Learning

References

1.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin. (2025). Attention Is All You Need. , 30, 5998–6008[10.65215/2q58a426]

2.Francisco Ordóñez, Daniel Roggen. (2016). Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors, 16(1), 115[10.3390/s16010115]

3.Attila Reiss, Didier Stricker. (2012). Introducing a New Benchmarked Dataset for Activity Monitoring. , 108–109[10.1109/iswc.2012.13]

4.Jubai An. (2015). Variational autoencoder based anomaly detection using reconstruction probability. , 2, 1

5.Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Lukowicz, David Bannach, Gerald Pirkl, Alois Ferscha, Jakob Doppler, Clemens Holzmann, Marc Kurz, Gerald Holl, Ricardo Chavarriaga, Hesam Sagha, Hamidreza Bayati, Marco Creatura, José del R. Millán. (2010). Collecting complex activity datasets in highly rich networked sensor environments. , 233–240[10.1109/inss.2010.5573462]

6.Marc Bächlin, Meir Plotnik, Daniel Roggen, Inbal Maidan, Jeffrey M. Hausdorff, Nir Giladi, Gerhard Tröster. (2009). Wearable Assistant for Parkinson’s Disease Patients With the Freezing of Gait Symptom. IEEE Transactions on Information Technology in Biomedicine, 14(2), 436–446[10.1109/titb.2009.2036165]

7.Mi Zhang, Alexander A. Sawchuk. (2012). USC-HAD. , 1036–1043[10.1145/2370216.2370438]

8.Yu Guan, Thomas Plötz. (2017). Ensembles of Deep LSTM Learners for Activity Recognition using Wearables. Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies, 1(2), 1–28[10.1145/3090076]

9.Thomas Stiefmeier, Daniel Roggen, Georg Ogris, Paul Lukowicz, Gerhard Tr. (2008). Wearable Activity Tracking in Car Manufacturing. IEEE Pervasive Computing, 7(2), 42–50[10.1109/mprv.2008.40]

10.Haojie Ma, Wenzhong Li, Xiao Zhang, Songcheng Gao, Sanglu Lu. (2019). AttnSense: Multi-level Attention Mechanism For Multimodal Human Activity Recognition. , 3109–3115[10.24963/ijcai.2019/431]

11.Ming Zeng, Haoxiang Gao, Tong Yu, Ole J. Mengshoel, Helge Langseth, Ian Lane, Xiaobing Liu. (2018). Understanding and improving recurrent networks for human activity recognition by continuous attention. , 56–63[10.1145/3267242.3267286]

12.Liangying Peng, Ling Chen, Zhenan Ye, Yi Zhang. (2018). AROMA. Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies, 2(2), 1–16[10.1145/3214277]

13.Lili Meng, Bo Zhao, Bo Chang, Gao Huang, Wei Sun, Frederick Tung, Leonid Sigal. (2019). Interpretable Spatio-Temporal Attention for Video Action Recognition. , 1513–1522[10.1109/iccvw.2019.00189]

14.Yegang Du, Yuto Lim, Yasuo Tan. (2019). A Novel Human Activity Recognition and Prediction in Smart Home Based on Interaction. Sensors, 19(20), 4474[10.3390/s19204474]

15.Harish Haresamudram, David V. Anderson, Thomas Plötz. (2019). On the role of features in human activity recognition. , 78–88[10.1145/3341163.3347727]

16.Shang Gao, John X. Qiu, Mohammed Alawad, Jacob Hinkle, Noah Schaefferkoetter, Hong‐Jun Yoon, Blair Christian, Paul Fearn, Lynne Penberthy, Xiao‐Cheng Wu, Linda Coyle, Georgia D. Tourassi, Arvind Ramanathan. (2019). Classifying cancer pathology reports with hierarchical self-attention networks. Artificial Intelligence in Medicine, 101, 101726[10.1016/j.artmed.2019.101726]

17.Hangwei Qian, Sinno Jialin Pan, Bingshui Da, Chunyan Miao. (2019). A Novel Distribution-Embedded Neural Network for Sensor-Based Activity Recognition. , 5614–5620[10.24963/ijcai.2019/779]

18.Saif Mahmud, M Tanjid Hasan Tonmoy, Kumar Bhaumik Kishor, A K M Mahbubur Rahman, M. Ashraful Amin, Mohammad Shoyaib, Asif Hossain Khan Muhammad, Ahsan Ali Amin. (2020). Human Activity Recognition from Wearable Sensor Data Using Self-Attention. Frontiers in artificial intelligence and applications, 1332–1339[10.3233/faia200236]

19.Shuochao Yao, Yiran Zhao, Huajie Shao, Dongxin Liu, Shengzhong Liu, Yifan Hao, Ailing Piao, Shaohan Hu, Lü Su, Tarek Abdelzaher. (2019). SADeepSense: Self-Attention Deep Learning Framework for Heterogeneous On-Device Sensors in Internet of Things Applications. , 1243–1251[10.1109/infocom.2019.8737500]

20.Fadi Al Machot, Mohammed R. Elkobaisi, Kyandoghere Kyamakya. (2020). Zero-Shot Human Activity Recognition Using Non-Visual Sensors. Sensors, 20(3), 825[10.3390/s20030825]

21.Hyeokhyen Kwon, Gregory D. Abowd, Thomas Plötz. (2018). Adding structural characteristics to distribution-based accelerometer representations for activity recognition using wearables. , 72–75[10.1145/3267242.3267258]

22.Vishvak Murahari, Thomas Ploetz. (2018). On attention models for human activity recognition. , 100–103[10.1145/3267242.3267287]

23.Weihao Cheng, Sarah Erfani, Rui Zhang, Kotagiri Ramamohanarao. (2018). Predicting Complex Activities from Ongoing Multivariate Time Series. , 3322–3328[10.24963/ijcai.2018/461]

24.Zengwei Zheng, Lifei Shi, Chi Wang, Lin Sun, Gang Pan. (2019). LSTM with Uniqueness Attention for Human Activity Recognition. Lecture notes in computer science, 498–509[10.1007/978-3-030-30508-6_40]

25.Kandan Ramakrishnan, Mathew Monfort, Barry McNamara, Alex Lascelles, Dan Gutfreund, Rogério Feris, Aude Oliva. (2019). Identifying Interpretable Action Concepts in Deep Networks. Computer Vision and Pattern Recognition, 12–15

Cited By

1.Manuel Gil-Martín, Rubén San-Segundo, Fernando Fernández-Martínez, Javier Ferreiros-López. (2021). Time Analysis in Human Activity Recognition. Neural Processing Letters, 53(6), 4507–4525[10.1007/s11063-021-10611-w]

2.Marcel Dix, Jan Jens Koltermann, Sebastian Mieck, Heiko Petersen, Sebastian Taege, Gahana Anjanappa. (2024). Using Siamese Neural Networks for the Open Set Recognition of Anomalies Detected in Industrial Time Series Data. , 1–8[10.1109/etfa61755.2024.10710747]