Assessment of rehabilitation exercises from depth sensor data

Shehzan Haider Chowdhury; Murshed Al Amin; AKM Mahbubur Rahman; M Ashraful Amin; Amin Ahsan Ali

doi:10.1109/iccit54785.2021.9689826

Conference Paper2021Cited by 10

Assessment of rehabilitation exercises from depth sensor data

Shehzan Haider Chowdhury, Murshed Al Amin, AKM Mahbubur Rahman, M Ashraful Amin, Amin Ahsan Ali

2021 24th International Conference on Computer and Information Technology (ICCIT)

IEEE, pp. 1–7

DOI: 10.1109/iccit54785.2021.9689826 URL

Abstract

Assessing the rehabilitation exercises are essential in the recovery and treatment of various musculoskeletal conditions following surgery. According to reports, over 90% of all rehabilitative exercise sessions are conducted in a home environment. As the number of patients grows, this method becomes prohibitively expensive. Providing technology support for home-based rehabilitation is an excellent approach to address this. The patient remains at home and does the exercises in front of the camera, with the footage or data being sent to the physician for comments on the exercises. In this paper, we propose two machine learning-based models to assess the quality of exercises where the data is captured by such kinect 3D sensors. The proposed models consist of a long short-term memory(LSTM) network which uses the time series skeletal data to predict the quality of the exercises. The first model uses the predefined features proposed by the physicians. For the second model, we extract features using graph convolutional network(GCN) on the skeletal data where each node represents a body part or joint in the body and the edges represent the connection between the body parts. We conclude that LSTM is more accurate at predicting the results when GCN features are used.

Keywords

Computer scienceRehabilitationGraphNode (physics)Machine learningArtificial intelligencePhysical medicine and rehabilitation

CCDS Authors

AKM Mahbubur Rahman, PhD

Director, DS & IPT Wings

Associate Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Prof. M Ashraful Amin, PhD

Founder Director, CCDS
Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

Amin Ahsan Ali, PhD

Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Research Interests

Artificial Intelligence, Machine Learning

References

1.Alex Graves, Jürgen Schmidhuber. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5-6), 602–610[10.1016/j.neunet.2005.06.042]
2.Ting Yao, Yingwei Pan, Yehao Li, Tao Mei. (2018). Exploring Visual Relationship for Image Captioning. Lecture notes in computer science, 711–727[10.1007/978-3-030-01264-9_42]
3.Hamed Pirsiavash, Carl Vondrick, Antonio Torralba. (2014). Assessing the Quality of Actions. Lecture notes in computer science, 556–571[10.1007/978-3-319-10599-4_36]
4.Yalin Liao, Aleksandar Vakanski, Min Xian. (2020). A Deep Learning Framework for Assessing Physical Rehabilitation Exercises. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(2), 468–477[10.1109/tnsre.2020.2966249]
5.Aleksandar Vakanski, Hyung-pil Jun, David R. Paul, Russell T. Baker. (2018). A Data Set of Human Body Movements for Physical Rehabilitation Exercises. Data, 3(1), 2[10.3390/data3010002]
6.Marianna Capecci, Maria Gabriella Ceravolo, Francesco Ferracuti, Sabrina Iarlori, Andrea Monteriù, Luca Romeo, Federica Verdini. (2019). The KIMORE Dataset: KInematic Assessment of MOvement and Clinical Scores for Remote Monitoring of Physical REhabilitation. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27(7), 1436–1448[10.1109/tnsre.2019.2923060]
7.Rui Zhao, Kang Wang, Hui Su, Qiang Ji. (2019). Bayesian Graph Convolution LSTM for Skeleton Based Action Recognition. , 6881–6891[10.1109/iccv.2019.00698]
8.Zhishuai Li, Gang Xiong, Yuanyuan Chen, Yisheng Lv, Bin Hu, Fenghua Zhu, Fei‐Yue Wang. (2019). A Hybrid Deep Learning Approach with GCN and LSTM for Traffic Flow Prediction. , 1929–1933[10.1109/itsc.2019.8916778]
9.Amr Elkholy, Mohamed E. Hussein, Walid Gomaa, Dima Damen, Emmanuel Kamal Aziz Saba. (2019). Efficient and Robust Skeleton-Based Quality Assessment and Abnormality Detection in Human Action Performance. IEEE Journal of Biomedical and Health Informatics, 24(1), 280–291[10.1109/jbhi.2019.2904321]
10.Sandra Bassett, Harry Prapavessis. (2007). Home-Based Physical Therapy Intervention With Adherence-Enhancing Strategies Versus Clinic-Based Management for Patients With Ankle Sprains. Physical Therapy, 87(9), 1132–1143[10.2522/ptj.20060260]
11.Steven R. Machlin, Julia Chevan, William Yu, Marc W. Zodet. (2011). Determinants of Utilization and Expenditures for Episodes of Ambulatory Physical Therapy Among Adults. Physical Therapy, 91(7), 1018–1029[10.2522/ptj.20100343]
12.Ravi Komatireddy. (2014). Quality and Quantity of Rehabilitation Exercises Delivered By A 3-D Motion Controlled Camera: A Pilot Study. International Journal of Physical Medicine & Rehabilitation, 02(04)[10.4172/2329-9096.1000214]
13.Faegheh Sardari. (2020). Bristol Research (University of Bristol)[10.3390/s20185258]
14.Ben Crabbe, Adeline Paiement, Sion Hannuna, Majid Mirmehdi. (2015). Skeleton-Free Body Pose Estimation from Depth Images for Movement Analysis. , 312–320[10.1109/iccvw.2015.49]
15.Han Zhang, Yonghong Song, Yuanlin Zhang. (2019). Graph Convolutional LSTM Model for Skeleton-Based Action Recognition. , 412–417[10.1109/icme.2019.00078]
16.Yanguo Huang, Shuo Zhang, Junlin Wen, Xinqiang Chen. (2020). Short-Term Traffic Flow Prediction Based on Graph Convolutional Network Embedded LSTM. International Conference on Transportation and Development 2020, 159–168[10.1061/9780784483152.014]
17.Fadilla ‘Atyka Nor Rashid, Nor Surayahani Suriani. (2020). Spiking neural network classification for spike train analysis of physiotherapy movements. Bulletin of Electrical Engineering and Informatics, 9(1), 319–325[10.11591/eei.v9i1.1868]
18.Benedek Rózemberczki, Paul Scherer, Yixuan He, George Panagopoulos, Alexander Riedel, Maria Astefanoaei, Olivér Kiss, Ferenc Béres, Guzmán López, Nicolas Collignon, Rik Sarkar. Edinburgh Research Explorer (University of Edinburgh)[link]
19.Jinyin Chen, Wang, Xueke, Xuanheng Xu. (2018). GC-LSTM: Graph Convolution Embedded LSTM for Dynamic Link Prediction. arXiv (Cornell University)[10.48550/arxiv.1812.04206]

Cited By

1.Fotos Frangoudes, Maria Matsangidou, Eirini C. Schiza, Kleanthis Neokleous, Constantinos S. Pattichis. (2022). Assessing Human Motion During Exercise Using Machine Learning: A Literature Review. IEEE Access, 10, 86874–86903[10.1109/access.2022.3198935]
2.Kevin Reby, Idriss Dulau, Guillaume Dubrasquet, Marie Beurton Aimar. (2023). Graph Transformer for Physical Rehabilitation Evaluation. , 1–8[10.1109/fg57933.2023.10042778]
3.Mohamed Jleli, Bessem Samet, Ashit Kumar Dutta. (2024). Artificial Intelligence-driven Remote Monitoring Model for Physical Rehabilitation. Journal of Disability Research, 3(1)[10.57197/jdr-2023-0065]
4.Büşra Açış, Selda Güney. (2022). Classification of human movements by using Kinect sensor. Biomedical Signal Processing and Control, 81, 104417[10.1016/j.bspc.2022.104417]
5.Alireza Ettefagh, Atena Roshan Fekr. (2024). Technological advances in lower-limb tele-rehabilitation: A review of literature. Journal of Rehabilitation and Assistive Technologies Engineering, 11, 20556683241259256[10.1177/20556683241259256]
6.Yanli Zhang, Tao Wang, Yan Wang, Jingyu Cao. (2025). Knowledge discovery of diseases symptoms and rehabilitation measures in Q&A communities. Scientific Reports, 15(1), 13593[10.1038/s41598-025-98300-9]
7.Tanawat Matangkasombut, Wuttipong Kumwilaisak, Chatchawarn Hansakunbuntheung, Nattanun Thatphithakkul. (2026). Spatial–Temporal Transformers With Stochastic Time-Warping and Joint-Wise Encoding for Rehabilitation Exercise Assessment. IEEE Open Journal of the Computer Society, 7, 190–201[10.1109/ojcs.2025.3650355]
8.Elham Mottaghi, Mohammad-R Akbarzadeh-T. (2024). Fuzzy Synchronization Likelihood Graph in Deep Neural Networks for Human Motion Time Series Analysis. IEEE Journal of Biomedical and Health Informatics, 29(1), 572–582[10.1109/jbhi.2024.3468899]
9.Weiwen Yang, Yingwei Zhang, Hai Liu, Yiqiang Chen. (2024). Leveraging Spatial-Temporal-Scale Augmentation to Assess Motor Abilities in Stroke Patients. , 3891–3896[10.1109/bibm62325.2024.10822320]
10.Bo Su, Kefei Shi. (2026). Body part-based dual-task spatio-temporal graph convolutional network for action quality assessment. International Journal of Machine Learning and Cybernetics, 17(8)[10.1007/s13042-026-03253-8]

← All publications

Abstract

CCDS Authors

AKM Mahbubur Rahman, PhD

Director, DS & IPT Wings

Associate Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Prof. M Ashraful Amin, PhD

Founder Director, CCDS
Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

Amin Ahsan Ali, PhD

Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Research Interests

Artificial Intelligence, Machine Learning

References

1.Alex Graves, Jürgen Schmidhuber. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5-6), 602–610[10.1016/j.neunet.2005.06.042]

2.Ting Yao, Yingwei Pan, Yehao Li, Tao Mei. (2018). Exploring Visual Relationship for Image Captioning. Lecture notes in computer science, 711–727[10.1007/978-3-030-01264-9_42]

3.Hamed Pirsiavash, Carl Vondrick, Antonio Torralba. (2014). Assessing the Quality of Actions. Lecture notes in computer science, 556–571[10.1007/978-3-319-10599-4_36]

4.Yalin Liao, Aleksandar Vakanski, Min Xian. (2020). A Deep Learning Framework for Assessing Physical Rehabilitation Exercises. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(2), 468–477[10.1109/tnsre.2020.2966249]

5.Aleksandar Vakanski, Hyung-pil Jun, David R. Paul, Russell T. Baker. (2018). A Data Set of Human Body Movements for Physical Rehabilitation Exercises. Data, 3(1), 2[10.3390/data3010002]

6.Marianna Capecci, Maria Gabriella Ceravolo, Francesco Ferracuti, Sabrina Iarlori, Andrea Monteriù, Luca Romeo, Federica Verdini. (2019). The KIMORE Dataset: KInematic Assessment of MOvement and Clinical Scores for Remote Monitoring of Physical REhabilitation. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27(7), 1436–1448[10.1109/tnsre.2019.2923060]

7.Rui Zhao, Kang Wang, Hui Su, Qiang Ji. (2019). Bayesian Graph Convolution LSTM for Skeleton Based Action Recognition. , 6881–6891[10.1109/iccv.2019.00698]

8.Zhishuai Li, Gang Xiong, Yuanyuan Chen, Yisheng Lv, Bin Hu, Fenghua Zhu, Fei‐Yue Wang. (2019). A Hybrid Deep Learning Approach with GCN and LSTM for Traffic Flow Prediction. , 1929–1933[10.1109/itsc.2019.8916778]

9.Amr Elkholy, Mohamed E. Hussein, Walid Gomaa, Dima Damen, Emmanuel Kamal Aziz Saba. (2019). Efficient and Robust Skeleton-Based Quality Assessment and Abnormality Detection in Human Action Performance. IEEE Journal of Biomedical and Health Informatics, 24(1), 280–291[10.1109/jbhi.2019.2904321]

10.Sandra Bassett, Harry Prapavessis. (2007). Home-Based Physical Therapy Intervention With Adherence-Enhancing Strategies Versus Clinic-Based Management for Patients With Ankle Sprains. Physical Therapy, 87(9), 1132–1143[10.2522/ptj.20060260]

11.Steven R. Machlin, Julia Chevan, William Yu, Marc W. Zodet. (2011). Determinants of Utilization and Expenditures for Episodes of Ambulatory Physical Therapy Among Adults. Physical Therapy, 91(7), 1018–1029[10.2522/ptj.20100343]

12.Ravi Komatireddy. (2014). Quality and Quantity of Rehabilitation Exercises Delivered By A 3-D Motion Controlled Camera: A Pilot Study. International Journal of Physical Medicine & Rehabilitation, 02(04)[10.4172/2329-9096.1000214]

13.Faegheh Sardari. (2020). Bristol Research (University of Bristol)[10.3390/s20185258]

14.Ben Crabbe, Adeline Paiement, Sion Hannuna, Majid Mirmehdi. (2015). Skeleton-Free Body Pose Estimation from Depth Images for Movement Analysis. , 312–320[10.1109/iccvw.2015.49]

15.Han Zhang, Yonghong Song, Yuanlin Zhang. (2019). Graph Convolutional LSTM Model for Skeleton-Based Action Recognition. , 412–417[10.1109/icme.2019.00078]

16.Yanguo Huang, Shuo Zhang, Junlin Wen, Xinqiang Chen. (2020). Short-Term Traffic Flow Prediction Based on Graph Convolutional Network Embedded LSTM. International Conference on Transportation and Development 2020, 159–168[10.1061/9780784483152.014]

17.Fadilla ‘Atyka Nor Rashid, Nor Surayahani Suriani. (2020). Spiking neural network classification for spike train analysis of physiotherapy movements. Bulletin of Electrical Engineering and Informatics, 9(1), 319–325[10.11591/eei.v9i1.1868]

18.Benedek Rózemberczki, Paul Scherer, Yixuan He, George Panagopoulos, Alexander Riedel, Maria Astefanoaei, Olivér Kiss, Ferenc Béres, Guzmán López, Nicolas Collignon, Rik Sarkar. Edinburgh Research Explorer (University of Edinburgh)[link]

19.Jinyin Chen, Wang, Xueke, Xuanheng Xu. (2018). GC-LSTM: Graph Convolution Embedded LSTM for Dynamic Link Prediction. arXiv (Cornell University)[10.48550/arxiv.1812.04206]

Cited By

1.Fotos Frangoudes, Maria Matsangidou, Eirini C. Schiza, Kleanthis Neokleous, Constantinos S. Pattichis. (2022). Assessing Human Motion During Exercise Using Machine Learning: A Literature Review. IEEE Access, 10, 86874–86903[10.1109/access.2022.3198935]

2.Kevin Reby, Idriss Dulau, Guillaume Dubrasquet, Marie Beurton Aimar. (2023). Graph Transformer for Physical Rehabilitation Evaluation. , 1–8[10.1109/fg57933.2023.10042778]

3.Mohamed Jleli, Bessem Samet, Ashit Kumar Dutta. (2024). Artificial Intelligence-driven Remote Monitoring Model for Physical Rehabilitation. Journal of Disability Research, 3(1)[10.57197/jdr-2023-0065]

4.Büşra Açış, Selda Güney. (2022). Classification of human movements by using Kinect sensor. Biomedical Signal Processing and Control, 81, 104417[10.1016/j.bspc.2022.104417]

5.Alireza Ettefagh, Atena Roshan Fekr. (2024). Technological advances in lower-limb tele-rehabilitation: A review of literature. Journal of Rehabilitation and Assistive Technologies Engineering, 11, 20556683241259256[10.1177/20556683241259256]

6.Yanli Zhang, Tao Wang, Yan Wang, Jingyu Cao. (2025). Knowledge discovery of diseases symptoms and rehabilitation measures in Q&A communities. Scientific Reports, 15(1), 13593[10.1038/s41598-025-98300-9]

7.Tanawat Matangkasombut, Wuttipong Kumwilaisak, Chatchawarn Hansakunbuntheung, Nattanun Thatphithakkul. (2026). Spatial–Temporal Transformers With Stochastic Time-Warping and Joint-Wise Encoding for Rehabilitation Exercise Assessment. IEEE Open Journal of the Computer Society, 7, 190–201[10.1109/ojcs.2025.3650355]

8.Elham Mottaghi, Mohammad-R Akbarzadeh-T. (2024). Fuzzy Synchronization Likelihood Graph in Deep Neural Networks for Human Motion Time Series Analysis. IEEE Journal of Biomedical and Health Informatics, 29(1), 572–582[10.1109/jbhi.2024.3468899]

9.Weiwen Yang, Yingwei Zhang, Hai Liu, Yiqiang Chen. (2024). Leveraging Spatial-Temporal-Scale Augmentation to Assess Motor Abilities in Stroke Patients. , 3891–3896[10.1109/bibm62325.2024.10822320]

10.Bo Su, Kefei Shi. (2026). Body part-based dual-task spatio-temporal graph convolutional network for action quality assessment. International Journal of Machine Learning and Cybernetics, 17(8)[10.1007/s13042-026-03253-8]