Conference Paper2024

UFlow-Net: A Unified Approach for Improved Video Frame Interpolation

Fatin Israq, Saadia Binte Alam, Halima Khatun, Shuvro Sourav Sarker, Siam Tahsin Bhuiyan, Mahmudul Haque, Rashedur Rahman, Syoji Kobashi

2024 27th International Conference on Computer and Information Technology (ICCIT)

IEEE, pp. 340-344

DOI: 10.1109/iccit64611.2024.11021809URL

Abstract

Video frame interpolation is a critical method in video processing, enabling the synthesis of intermediate frames to enhance video quality and frame rates. While conventional methods rely on optical flow estimation and kernel-based approaches, they often struggle with complex scenes and rapid motion. Recent advancements in deep learning have demonstrated the potential of hybrid architectures to address these challenges. This paper proposes UFlow-Net, a novel hybrid framework that integrates flow estimation and kernel-based methods within a modified U-Net architecture. The model incorporates a flow-enhanced encoder-decoder structure with advanced feature fusion via skip connections to preserve spatial-temporal information. Evaluated on the Vimeo90K, UCF101, and Middlebury datasets, UFlow-Net achieves state-of-the-art performance with peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) metrics. Results demonstrate its robustness in handling intricate backgrounds and dynamic motion, making it a significant advancement in the field of video frame interpolation.

Keywords

  • frame interpolation
  • optical flow
  • U-Net
  • feature fusion
  • skip connections
  • encoder-decoder architecture
  • dynamic motion

CCDS Authors

Saadia Binte Alam, PhD
Saadia Binte Alam, PhD

Saadia Binte Alam, PhD

  • Director, MIRA & WiSE Wings

Associate Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

HCII, Scientific Reports, Cognitive Computation, Neural Computing and Applications

Halima Khatun
Halima Khatun

Halima Khatun

  • Research Assistant, CCDS

Bachelors

Independent University, Bangladesh

Published at

Scientific Reports

Siam Tahsin Bhuiyan
Siam Tahsin Bhuiyan

Siam Tahsin Bhuiyan

  • Intern, CCDS

Bachelors

Independent University, Bangladesh

Published at

Scientific Reports

Md. Rashedur Rahman, D. Eng.
Md. Rashedur Rahman, D. Eng.

Md. Rashedur Rahman, D. Eng.

  • Co-Director, MIRA & NEST Wings

Assistant Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

CHI, HCII, Scientific Reports, Cognitive Computation, Neural Computing and Applications

References

  1. 1.Olaf Ronneberger, Philipp Fischer, Thomas Brox. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Lecture notes in computer science, 234–241[10.1007/978-3-319-24574-4_28]
  2. 2.Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox. (2017). FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 1647–1655[10.1109/cvpr.2017.179]
  3. 3.Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, Richard Szeliski. (2010). A Database and Evaluation Methodology for Optical Flow. International Journal of Computer Vision, 92(1), 1–31[10.1007/s11263-010-0390-2]
  4. 4.Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, William T. Freeman. (2019). Video Enhancement with Task-Oriented Flow. International Journal of Computer Vision, 127(8), 1106–1125[10.1007/s11263-018-01144-2]
  5. 5.Simon Niklaus, Long Mai, Feng Liu. (2017). Video Frame Interpolation via Adaptive Separable Convolution. IEEE International Conference on Computer Vision (ICCV), 261–270[10.1109/iccv.2017.37]
  6. 6.Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, Ming-Hsuan Yang. (2019). Depth-Aware Video Frame Interpolation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 3698–3707[10.1109/cvpr.2019.00382]
  7. 7.Simon Niklaus, Feng Liu. (2018). Context-Aware Synthesis for Video Frame Interpolation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 1701–1710[10.1109/cvpr.2018.00183]
  8. 8.Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, Ming-Hsuan Yang. (2019). MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(3), 933–948[10.1109/tpami.2019.2941941]
  9. 9.Chao-Yuan Wu, Nayan Singhal, Philipp Krähenbühl. (2018). Video Compression Through Image Interpolation. Lecture notes in computer science, 425–440[10.1007/978-3-030-01237-3_26]
  10. 10.Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, Alexander Sorkine-Hornung. (2015). Phase-based frame interpolation for video. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 1410–1418[10.1109/cvpr.2015.7298747]
  11. 11.Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, Shuchang Zhou. (2022). Real-Time Intermediate Flow Estimation for Video Frame Interpolation. Lecture notes in computer science, 624–642[10.1007/978-3-031-19781-9_36]
  12. 12.Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Daehyun Pak, Yuseok Ban, Sangyoun Lee. (2020). AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 5315–5324[10.1109/cvpr42600.2020.00536]
  13. 13.Simone Meyer, Abdelaziz Djelouah, Brian McWilliams, Alexander Sorkine-Hornung, Markus Gross, Christopher Schroers. (2018). PhaseNet for Video Frame Interpolation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 498–507[10.1109/cvpr.2018.00059]
  14. 14.Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang, Jie Yang. (2022). IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 1959–1968[10.1109/cvpr52688.2022.00201]
  15. 15.Xianhang Cheng, Zhenzhong Chen. (2021). Multiple Video Frame Interpolation via Enhanced Deformable Separable Convolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 7029–7045[10.1109/tpami.2021.3100714]
  16. 16.Xianhang Cheng, Zhenzhong Chen. (2020). Video Frame Interpolation via Deformable Separable Convolution. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07), 10607–10614[10.1609/aaai.v34i07.6634]
  17. 17.Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, Du Tran. (2023). FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2070–2081[10.1109/wacv56688.2023.00211]
  18. 18.Tianyu Ding, Luming Liang, Zhihui Zhu, Ilya Zharkov. (2021). CDFI: Compression-Driven Network Design for Frame Interpolation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 7997–8007[10.1109/cvpr46437.2021.00791]
  19. 19.Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, Zhiyong Gao. (2020). Blurry Video Frame Interpolation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 5113–5122[10.1109/cvpr42600.2020.00516]
  20. 20.Ping Hu, Simon Niklaus, Stan Sclaroff, Kate Saenko. (2022). Many-to-many Splatting for Efficient Video Frame Interpolation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 3543–3552[10.1109/cvpr52688.2022.00354]
  21. 21.Jiong Dong, Kaoru Ota, Mianxiong Dong. (2023). Video Frame Interpolation: A Comprehensive Survey. ACM Transactions on Multimedia Computing Communications and Applications, 19(2s), 1–31[10.1145/3556544]
  22. 22.Wenbo Bao, Xiaoyun Zhang, Li Chen, Lianghui Ding, Zhiyong Gao. (2018). High-Order Model and Dynamic Filtering for Frame Rate Up-Conversion. IEEE Transactions on Image Processing, 27(8), 3813–3826[10.1109/tip.2018.2825100]
  23. 23.Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, Zhiyong Gao. (2020). Video Frame Interpolation and Enhancement via Pyramid Recurrent Framework. IEEE Transactions on Image Processing, 30, 277–292[10.1109/tip.2020.3033617]
  24. 24.Minho Park, Hak Gu Kim, Sangmin Lee, Yong Man Ro. (2020). Robust Video Frame Interpolation With Exceptional Motion Map. IEEE Transactions on Circuits and Systems for Video Technology, 31(2), 754–764[10.1109/tcsvt.2020.2981964]
  25. 25.Jiale He, Gaobo Yang, Xin Liu, Xiangling Ding. (2020). Spatio-temporal Saliency-based Motion Vector Refinement for Frame Rate Up-conversion. ACM Transactions on Multimedia Computing Communications and Applications, 16(2), 1–18[10.1145/3382506]
  26. 26.Saikat Dutta, Arulkumar Subramaniam, Anurag Mittal. (2022). Non-linear Motion Estimation for Video Frame Interpolation using Space-time Convolutions. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1725–1730[10.1109/cvprw56347.2022.00180]
  27. 27.Xiaohui Yang, Haoran Zhang, Zhe Qu, Zhiquan Feng, Jinglan Tian. (2022). Video frame interpolation via residual blocks and feature pyramid networks. IET Image Processing, 17(4), 1060–1070[10.1049/ipr2.12695]
  28. 28.Soomro, Khurram, Zamir, Amir Roshan, Shah, Mubarak. (2012). UCF-101: A dataset of 101 human actions classes from videos in the wild. arXiv (Cornell University)[10.48550/arxiv.1212.0402]
  29. 29.Xue, Tianfan, Chen, Baian, Wu, Jiajun, Wei, Donglai, Freeman, William T. (2019). Video Enhancement with Task-Oriented Flow. DSpace@MIT (Massachusetts Institute of Technology)[link]
← All publications