Conference Paper2019Cited by 10

Beat Based Realistic Dance Video Generation using Deep Learning

Md Shazid Islam, Md Saydur Rahman, M Ashraful Amin

2019 IEEE International Conference on Robotics, Automation, Artificial-intelligence and Internet-of-Things (RAAICON)

IEEE, pp. 43–47

DOI: 10.1109/raaicon48939.2019.22URL

Abstract

Deep learning based feature extraction has enabled us to synchronize audio and body movements. It is a promising research field which has great applications in generating sign language, computer animations as well as dance. Previously, computer generated choreography was limited to just stick figure representation. This paper adds image translation technique in dance generation which produces realistic dance moves. With this technique it is possible to produce dance video of a amateur person dancing like a professional. The mapping of stick figure to realistic image is done using Generative Adversarial Network (GAN). We created our own dataset and after adversarial training reconstructed images have SSIM mean 0.864 and LPIPS mean 0.0168. This method produces realistic dance video which is beat based. Body movement speed varies according to the tempo of music which makes it more relevant to real life dance movement.

Keywords

  • generative adversarial networks
  • image-to-image translation
  • audio-video synchronization

CCDS Authors

Prof. M Ashraful Amin, PhD
M Ashraful Amin, PhD

Prof. M Ashraful Amin, PhD

  • Founder Director, CCDS
  • Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Published at

ACL, CHI, UbiComp, ECAI, WACV, PAKDD, IJCNN, ICPR, ICIP, HCII, CLEF, BLP, Scientific Reports, JMIR, IEEE Access, Smart Agricultural Technology, Frontiers in Computational Neuroscience, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

References

  1. 1.Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, Eero P. Simoncelli. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612[10.1109/tip.2003.819861]
  2. 2.Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, Oliver Wang. (2018). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. , 586–595[10.1109/cvpr.2018.00068]
  3. 3.Justin Johnson, Alexandre Alahi, Li Fei-Fei. (2016). Perceptual Losses for Real-Time Style Transfer and Super-Resolution. Lecture notes in computer science, 694–711[10.1007/978-3-319-46475-6_43]
  4. 4.Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh. (2017). Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields. , 1302–1310[10.1109/cvpr.2017.143]
  5. 5.Brian McFee, Colin Raffel, Dawen Liang, Daniel P. W. Ellis, Matt McVicar, Eric Battenberg, Oriol Nieto. (2015). librosa: Audio and Music Signal Analysis in Python. Proceedings of the Python in Science Conferences, 18–24[10.25080/majora-7b98e3ed-003]
  6. 6.Quan Huynh‐Thu, M. Ghanbari. (2008). Scope of validity of PSNR in image/video quality assessment. Electronics Letters, 44(13), 800–801[10.1049/el:20080522]
  7. 7.Caroline Chan, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros. (2019). Everybody Dance Now. , 5932–5941[10.1109/iccv.2019.00603]
  8. 8.Daniel P. W. Ellis. (2007). Beat Tracking by Dynamic Programming. Journal of New Music Research, 36(1), 51–60[10.1080/09298210701653344]
  9. 9.Masataka Goto. (2006). A chorus section detection method for musical audio signals and its application to a music listening station. IEEE Transactions on Audio Speech and Language Processing, 14(5), 1783–1794[10.1109/tsa.2005.863204]
  10. 10.Taoran Tang, Jia Jia, Hanyang Mao. (2018). Dance with Melody. , 1598–1606[10.1145/3240508.3240526]
  11. 11.Ferda Ofli, Engin Erzin, Y. Yemez, A. Murat Tekalp. (2011). Learn2Dance: Learning Statistical Music-to-Dance Mappings for Choreography Synthesis. IEEE Transactions on Multimedia, 14(3), 747–759[10.1109/tmm.2011.2181492]
  12. 12.Nelson Yalta, Shinji Watanabe, Kazuhiro Nakadai, Tetsuya Ogata. (2019). Weakly-Supervised Deep Recurrent Neural Networks for Basic Dance Step Generation. , 1–8[10.1109/ijcnn.2019.8851872]
  13. 13.Marwa A. Nasr, Mohammed Alrahmawy, A. S. Tolba. (2016). Multi-scale structural similarity index for motion detection. Journal of King Saud University - Computer and Information Sciences, 29(3), 399–409[10.1016/j.jksuci.2016.02.004]
  14. 14.Takaaki Shiratori, Atsushi Nakazawa, Katsushi Ikeuchi. (2006). Synthesizing dance performance using musical and motion features. , 3654–3659[10.1109/robot.2006.1642260]
  15. 15.Yi Wang, Zhiqiang Liu, Lizhu Zhou. (2005). Learning hierarchical non-parametric hidden Markov model of human motion. , 3315–3320 Vol. 6[10.1109/icmlc.2005.1527515]
  16. 16.Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, Oliver Wang. (2018). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. arXiv (Cornell University)[10.48550/arxiv.1801.03924]
  17. 17.Luka Crnkovic-Friis, Louise Crnkovic-Friis. (2016). Generative Choreography using Deep Learning. arXiv (Cornell University), 272–277[10.48550/arxiv.1605.06921]
  18. 18.Juheon Lee, Seohyun Kim, Kyogu Lee. (2018). Listen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder Network. arXiv (Cornell University)[10.48550/arxiv.1811.00818]

Cited By

  1. 1.A S M Jahid Hasan, Md Saydur Rahman, Md Shazid Islam, Jubair Yusuf. (2023). Data Driven Energy Theft Localization in a Distribution Network. , 388–392[10.1109/icict4sd59951.2023.10303520]
  2. 2.A S M Jahid Hasan, Jubair Yusuf, Md Saydur Rahman, Md Shazid Islam. (2023). Electricity Cost Optimization for Large Loads through Energy Storage and Renewable Energy. , 46–50[10.1109/icict4sd59951.2023.10303409]
  3. 3.Md Saiful Islam Sajol, A S M Jahid Hasan, Md Shazid Islam, Md Saydur Rahman. (2024). A ConvNeXt V2 Approach to Document Image Analysis: Enhancing High-Accuracy Classification. , 1–6[10.1109/citds62610.2024.10791343]
  4. 4.Md Shazid Islam, A S M Jahid Hasan, Md Saydur Rahman, Jubair Yusuf, Md Saiful Islam Sajol, Farhana Akter Tumpa. (2023). Location Agnostic Source-Free Domain Adaptive Learning to Predict Solar Power Generation. , 1–6[10.1109/etfg55873.2023.10407265]
  5. 5.Md Saiful Islam Sajol, Jahid Hasan, Shazid Islam, Saydur Rahman. (2024). A ConvNeXt V2 Approach to Document Image Analysis: Enhancing High-Accuracy Classification. [10.36227/techrxiv.172254255.56093481/v1]
  6. 6.Md Saydur Rahman, Farhana Akter Tumpa, Md Shazid Islam, Abul Al Arabi, Md Sanzid Bin Hossain, Md Saad Ul Haque. (2023). Comparative Evaluation of Weather Forecasting using Machine Learning Models. , 1–6[10.1109/iccit60459.2023.10441077]
  7. 7.Md Shazid Islam, Md Saydur Rahman, Md Saad Ul Haque, Farhana Akter Tumpa, Md Sanzid Bin Hossain, Abul Al Arabi. (2023). Location Agnostic Adaptive Rain Precipitation Prediction using Deep Learning. , 148–153[10.1109/wiecon-ece60392.2023.10456479]
  8. 8.Md. Sanzid Bin Hossain, Md. Shazid Islam, Md. Saad Ul Haque, Md. Saydur Rahman. (2024). Gait Phase Classification from sEMG in Multiple Locomotion Mode Using Deep Learning. Lecture notes in networks and systems, 371–383[10.1007/978-981-97-3562-4_29]
  9. 9.Md Shazid Islam, S. M. Mahbubur Rahman. (2025). Synthesis of style-specific dance by learning body gestures from music. Multimedia Tools and Applications, 84(41), 49585–49606[10.1007/s11042-025-21123-3]
  10. 10.Sabbir Ahmed, Farhana Akter Tumpa, Sabiha Benta Sayed Badhon, Lamia Anjum. (2024). Video Based Vehicle Detection and Tracking using Image Processing and Deep Learning. , 14, 1–5[10.1109/icaeee62219.2024.10561808]
← All publications