Conference Paper2021Cited by 16

Assessment of rehabilitation exercises from depth sensor data

Shehzan Haider Chowdhury, Murshed Al Amin, AKM Mahbubur Rahman, M Ashraful Amin, Amin Ahsan Ali

2021 24th International Conference on Computer and Information Technology (ICCIT)

IEEE, pp. 1–7

  • Good Health and Well-being — Relevance: 93%
DOI: 10.1109/iccit54785.2021.9689826URL

Abstract

Assessing the rehabilitation exercises are essential in the recovery and treatment of various musculoskeletal conditions following surgery. According to reports, over 90% of all rehabilitative exercise sessions are conducted in a home environment. As the number of patients grows, this method becomes prohibitively expensive. Providing technology support for home-based rehabilitation is an excellent approach to address this. The patient remains at home and does the exercises in front of the camera, with the footage or data being sent to the physician for comments on the exercises. In this paper, we propose two machine learning-based models to assess the quality of exercises where the data is captured by such kinect 3D sensors. The proposed models consist of a long short-term memory(LSTM) network which uses the time series skeletal data to predict the quality of the exercises. The first model uses the predefined features proposed by the physicians. For the second model, we extract features using graph convolutional network(GCN) on the skeletal data where each node represents a body part or joint in the body and the edges represent the connection between the body parts. We conclude that LSTM is more accurate at predicting the results when GCN features are used.

Keywords

  • graph convolutional networks
  • home-based rehabilitation
  • skeleton data
  • long short-term memory networks
  • Kinect depth sensor
  • Feature extraction
  • musculoskeletal rehabilitation

CCDS Authors

AKM Mahbubur Rahman, PhD
AKM Mahbubur Rhman, PhD

AKM Mahbubur Rahman, PhD

  • Director, DS & IPT Wings

  • Associate Professor

    Department of Computer Science and Engineering

    Independent University, Bangladesh (IUB)

Published at

CHI, ECCV, WACV, PAKDD, LREC, IJCNN, ICPR, ICIP, HCII, BEA, Scientific Reports, Information Sciences

Prof. M Ashraful Amin, PhD
M Ashraful Amin, PhD

Prof. M Ashraful Amin, PhD

  • Founder Director, CCDS
  • Director, AI/ML, HCI & DS Wings

Published at

ACL, CHI, UbiComp, ECAI, WACV, PAKDD, IJCNN, ICPR, ICIP, HCII, CLEF, BLP, Scientific Reports, Pattern Recognition, JMIR, IEEE Access, Smart Agricultural Technology, Frontiers in Computational Neuroscience, Diabetes Research and Clinical Practice, PLOS ONE

Prof. Amin Ahsan Ali, PhD
Amin Ahsan Ali, PhD

Prof. Amin Ahsan Ali, PhD

  • Director, AI/ML Wing
  • Supervisor, ComBio Wing

  • Professor

    Department of Computer Science and Engineering

    Independent University, Bangladesh (IUB)

Published at

ACL, CHI, UbiComp, ECCV, IJCAI, ECAI, WACV, IPSN, PAKDD, LREC, IJCNN, ICPR, ICIP, BEA, BLP, ICMLA, Scientific Reports, Scientific Data, Pattern Recognition, Information Sciences, Ad Hoc Networks, Drug and Alcohol Dependence, PLOS ONE

References

  1. 1.Alex Graves, Jürgen Schmidhuber. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5-6), 602–610[10.1016/j.neunet.2005.06.042]
  2. 2.Ting Yao, Yingwei Pan, Yehao Li, Tao Mei. (2018). Exploring Visual Relationship for Image Captioning. Lecture notes in computer science, 711–727[10.1007/978-3-030-01264-9_42]
  3. 3.Hamed Pirsiavash, Carl Vondrick, Antonio Torralba. (2014). Assessing the Quality of Actions. Lecture notes in computer science, 556–571[10.1007/978-3-319-10599-4_36]
  4. 4.Yalin Liao, Aleksandar Vakanski, Min Xian. (2020). A Deep Learning Framework for Assessing Physical Rehabilitation Exercises. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(2), 468–477[10.1109/tnsre.2020.2966249]
  5. 5.Aleksandar Vakanski, Hyung-pil Jun, David R. Paul, Russell T. Baker. (2018). A Data Set of Human Body Movements for Physical Rehabilitation Exercises. Data, 3(1), 2[10.3390/data3010002]
  6. 6.Marianna Capecci, Maria Gabriella Ceravolo, Francesco Ferracuti, Sabrina Iarlori, Andrea Monteriù, Luca Romeo, Federica Verdini. (2019). The KIMORE Dataset: KInematic Assessment of MOvement and Clinical Scores for Remote Monitoring of Physical REhabilitation. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27(7), 1436–1448[10.1109/tnsre.2019.2923060]
  7. 7.Rui Zhao, Kang Wang, Hui Su, Qiang Ji. (2019). Bayesian Graph Convolution LSTM for Skeleton Based Action Recognition. , 6881–6891[10.1109/iccv.2019.00698]
  8. 8.Zhishuai Li, Gang Xiong, Yuanyuan Chen, Yisheng Lv, Bin Hu, Fenghua Zhu, Fei‐Yue Wang. (2019). A Hybrid Deep Learning Approach with GCN and LSTM for Traffic Flow Prediction. , 1929–1933[10.1109/itsc.2019.8916778]
  9. 9.Amr Elkholy, Mohamed E. Hussein, Walid Gomaa, Dima Damen, Emmanuel Kamal Aziz Saba. (2019). Efficient and Robust Skeleton-Based Quality Assessment and Abnormality Detection in Human Action Performance. IEEE Journal of Biomedical and Health Informatics, 24(1), 280–291[10.1109/jbhi.2019.2904321]
  10. 10.Sandra Bassett, Harry Prapavessis. (2007). Home-Based Physical Therapy Intervention With Adherence-Enhancing Strategies Versus Clinic-Based Management for Patients With Ankle Sprains. Physical Therapy, 87(9), 1132–1143[10.2522/ptj.20060260]
  11. 11.Steven R. Machlin, Julia Chevan, William Yu, Marc W. Zodet. (2011). Determinants of Utilization and Expenditures for Episodes of Ambulatory Physical Therapy Among Adults. Physical Therapy, 91(7), 1018–1029[10.2522/ptj.20100343]
  12. 12.Ravi Komatireddy. (2014). Quality and Quantity of Rehabilitation Exercises Delivered By A 3-D Motion Controlled Camera: A Pilot Study. International Journal of Physical Medicine & Rehabilitation, 02(04)[10.4172/2329-9096.1000214]
  13. 13.Faegheh Sardari. (2020). Bristol Research (University of Bristol)[10.3390/s20185258]
  14. 14.Ben Crabbe, Adeline Paiement, Sion Hannuna, Majid Mirmehdi. (2015). Skeleton-Free Body Pose Estimation from Depth Images for Movement Analysis. , 312–320[10.1109/iccvw.2015.49]
  15. 15.Han Zhang, Yonghong Song, Yuanlin Zhang. (2019). Graph Convolutional LSTM Model for Skeleton-Based Action Recognition. , 412–417[10.1109/icme.2019.00078]
  16. 16.Yanguo Huang, Shuo Zhang, Junlin Wen, Xinqiang Chen. (2020). Short-Term Traffic Flow Prediction Based on Graph Convolutional Network Embedded LSTM. International Conference on Transportation and Development 2020, 159–168[10.1061/9780784483152.014]
  17. 17.Fadilla ‘Atyka Nor Rashid, Nor Surayahani Suriani. (2020). Spiking neural network classification for spike train analysis of physiotherapy movements. Bulletin of Electrical Engineering and Informatics, 9(1), 319–325[10.11591/eei.v9i1.1868]
  18. 18.Benedek Rózemberczki, Paul Scherer, Yixuan He, George Panagopoulos, Alexander Riedel, Maria Astefanoaei, Olivér Kiss, Ferenc Béres, Guzmán López, Nicolas Collignon, Rik Sarkar. Edinburgh Research Explorer (University of Edinburgh)[link]
  19. 19.Jinyin Chen, Wang, Xueke, Xuanheng Xu. (2018). GC-LSTM: Graph Convolution Embedded LSTM for Dynamic Link Prediction. arXiv (Cornell University)[10.48550/arxiv.1812.04206]

Cited By

  1. 1.Sara Sardari, Sara Sharifzadeh, Alireza Daneshkhah, Bahareh Nakisa, Seng W. Loke, Vasile Palade, Michael J. Duncan. (2023). Artificial Intelligence for skeleton-based physical rehabilitation action evaluation: A systematic review. Computers in Biology and Medicine, 158, 106835[10.1016/j.compbiomed.2023.106835]
  2. 2.Youssef Mourchid, Rim Slama. (2023). D-STGCNT: A Dense Spatio-Temporal Graph Conv-GRU Network based on transformer for assessment of patient physical rehabilitation. Computers in Biology and Medicine, 165, 107420[10.1016/j.compbiomed.2023.107420]
  3. 3.Fotos Frangoudes, Maria Matsangidou, Eirini C. Schiza, Kleanthis Neokleous, Constantinos S. Pattichis. (2022). Assessing Human Motion During Exercise Using Machine Learning: A Literature Review. IEEE Access, 10, 86874–86903[10.1109/access.2022.3198935]
  4. 4.Kevin Reby, Idriss Dulau, Guillaume Dubrasquet, Marie Beurton Aimar. (2023). Graph Transformer for Physical Rehabilitation Evaluation. International Conference on Automatic Face and Gesture Recognition, 1–8[10.1109/fg57933.2023.10042778]
  5. 5.Sara Sardari, Sara Sharifzadeh, Alireza Daneshkhah, Seng W. Loke, Vasile Palade, Michael J. Duncan, Bahareh Nakisa. (2024). LightPRA: A Lightweight Temporal Convolutional Network for Automatic Physical Rehabilitation Exercise Assessment. Computers in Biology and Medicine, 173, 108382[10.1016/j.compbiomed.2024.108382]
  6. 6.Mohamed Jleli, Bessem Samet, Ashit Kumar Dutta. (2024). Artificial Intelligence-driven Remote Monitoring Model for Physical Rehabilitation. Journal of Disability Research, 3(1)[10.57197/jdr-2023-0065]
  7. 7.Büşra Açış, Selda Güney. (2022). Classification of human movements by using Kinect sensor. Biomedical Signal Processing and Control, 81, 104417[10.1016/j.bspc.2022.104417]
  8. 8.Ikram Kourbane, Panagiotis Papadakis, Mihai Andries. (2025). Optimized assessment of physical rehabilitation exercises using spatiotemporal, sequential graph-convolutional networks. Computers in Biology and Medicine, 186, 109578[10.1016/j.compbiomed.2024.109578]
  9. 9.Alireza Ettefagh, Atena Roshan Fekr. (2024). Technological advances in lower-limb tele-rehabilitation: A review of literature. Journal of Rehabilitation and Assistive Technologies Engineering, 11, 20556683241259256[10.1177/20556683241259256]
  10. 10.Ikram Kourbane, Panagiotis Papadakis, Mihai Andries. (2024). SSL-Rehab: Assessment of physical rehabilitation exercises through self-supervised learning of 3D skeleton representations. Computer Vision and Image Understanding, 251, 104275[10.1016/j.cviu.2024.104275]
  11. 11.Yanli Zhang, Tao Wang, Yan Wang, Jingyu Cao. (2025). Knowledge discovery of diseases symptoms and rehabilitation measures in Q&A communities. Scientific Reports, 15(1), 13593[10.1038/s41598-025-98300-9]
  12. 12.Md. Johir Raihan, Md. Golam Mustafa, Ali Saleh Alammary, Abdullah-Al Nahid. (2025). Rehabilitation exercise score prediction through integration of spatial feature with graph structural learning. Engineering Applications of Artificial Intelligence, 155, 110833[10.1016/j.engappai.2025.110833]
  13. 13.Bo Su, Kefei Shi. (2026). Body part-based dual-task spatio-temporal graph convolutional network for action quality assessment. International Journal of Machine Learning and Cybernetics, 17(8)[10.1007/s13042-026-03253-8]
  14. 14.Tanawat Matangkasombut, Wuttipong Kumwilaisak, Chatchawarn Hansakunbuntheung, Nattanun Thatphithakkul. (2026). Spatial–Temporal Transformers With Stochastic Time-Warping and Joint-Wise Encoding for Rehabilitation Exercise Assessment. IEEE Open Journal of the Computer Society, 7, 190–201[10.1109/ojcs.2025.3650355]
  15. 15.Weiwen Yang, Yingwei Zhang, Hai Liu, Yiqiang Chen. (2024). Leveraging Spatial-Temporal-Scale Augmentation to Assess Motor Abilities in Stroke Patients. IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 3891–3896[10.1109/bibm62325.2024.10822320]
  16. 16.Elham Mottaghi, Mohammad-R Akbarzadeh-T. (2024). Fuzzy Synchronization Likelihood Graph in Deep Neural Networks for Human Motion Time Series Analysis. IEEE Journal of Biomedical and Health Informatics, 29(1), 572–582[10.1109/jbhi.2024.3468899]
← All publications