Journal Article2022Cited by 14

Supplementary Material for Variational Stacked Local Attention Networks for Diverse Video Captioning

Tonmoay Deb, Akib Sadmanee, Kishor Kumar Bhaumik, Amin Ahsan Ali, M Ashraful Amin, AKM Mahbubur Rahman

2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

IEEE, pp. 2493-2502

DOI: 10.1109/wacv51458.2022.00255URL

Abstract

While describing spatiotemporal events in natural language, video captioning models mostly rely on the en-coder’s latent visual representation. Recent progress on the encoder-decoder model attends encoder features mainly in linear interaction with the decoder. However, growing model complexity for visual data encourages more explicit feature interaction for fine-grained information, which is currently absent in the video captioning domain. Moreover, feature aggregations methods have been used to un-veil richer visual representation, either by the concatenation or using a linear layer. Though feature sets for a video semantically overlap to some extent, these approaches result in objective mismatch and feature redundancy. In addition, diversity in captions is a fundamental component of expressing one event from several meaningful perspectives, currently missing in the temporal, i.e., video captioning domain. To this end, we propose Variational Stacked Local Attention Network (VSLAN), which exploits low-rank bilinear pooling for self-attentive feature interaction and stacking multiple video feature streams in a discount fashion. Each feature stack’s learned attributes contribute to our proposed diversity encoding module, followed by the decoding query stage to facilitate end-to-end diverse and natural captions without any explicit supervision on attributes. We evaluate VSLAN on MSVD and MSR-VTT datasets in terms of syntax and diversity. The CIDEr score of VSLAN outperforms current off-the-shelf methods by 7.8% on MSVD and 4.5% on MSR-VTT, respectively. On the same datasets, VSLAN achieves competitive results in caption diversity metrics.

Keywords

  • MSR-VTT
  • video captioning
  • caption diversity
  • feature interaction
  • self-attention
  • MSVD
  • feature aggregation
  • encoder-decoder architecture

CCDS Authors

Tonmoay Deb
Tonmoay Deb

Tonmoay Deb

  • Research Assistant, CCDS

PhD Student

Northwestern University

Published at

WACV

Kishor Kumar Bhaumik
Kishor Kumar Bhaumik

Kishor Kumar Bhaumik

PhD student

University of California, Riverside

Published at

WACV, Pattern Recognition

Research Interests

Machine Learning

Amin Ahsan Ali, PhD
Amin Ahsan Ali, PhD

Amin Ahsan Ali, PhD

  • Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

ACL, CHI, UbiComp, ECCV, IJCAI, ECAI, WACV, IPSN, PAKDD, LREC, IJCNN, ICPR, ICIP, BEA, BLP, ICMLA, Scientific Reports, Scientific Data, Pattern Recognition, Information Sciences, Ad Hoc Networks, Drug and Alcohol Dependence, PLOS ONE

Research Interests

Artificial Intelligence, Machine Learning

Prof. M Ashraful Amin, PhD
M Ashraful Amin, PhD

Prof. M Ashraful Amin, PhD

  • Founder Director, CCDS
  • Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Published at

ACL, CHI, UbiComp, ECAI, WACV, PAKDD, IJCNN, ICPR, ICIP, HCII, CLEF, BLP, Scientific Reports, JMIR, IEEE Access, Smart Agricultural Technology, Frontiers in Computational Neuroscience, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

AKM Mahbubur Rahman, PhD
AKM Mahbubur Rhman, PhD

AKM Mahbubur Rahman, PhD

  • Director, DS & IPT Wings

Associate Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

CHI, ECCV, WACV, PAKDD, LREC, IJCNN, ICPR, ICIP, HCII, BEA, Scientific Reports, Information Sciences

References

  1. 1.Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun. (2016). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149[10.1109/tpami.2016.2577031]
  2. 2.Yinhan Liu, Myle Ott, Arora, Ravneet, Jingfei Du. (2019). HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case. DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)[10.4230/lipics.cp.2025.31]
  3. 3.Jie Hu, Li Shen, Samuel Albanie, Gang Sun, Enhua Wu. (2019). Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(8), 2011–2023[10.1109/tpami.2019.2913372]
  4. 4.Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri. (2015). Learning Spatiotemporal Features with 3D Convolutional Networks. , 4489–4497[10.1109/iccv.2015.510]
  5. 5.Kensho Hara, Hirokatsu Kataoka, Yutaka Satoh. (2018). Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?. , 6546–6555[10.1109/cvpr.2018.00685]
  6. 6.Jun Xu, Tao Mei, Ting Yao, Yong Rui. (2016). MSR-VTT: A Large Video Description Dataset for Bridging Video and Language. , 5288–5296[10.1109/cvpr.2016.571]
  7. 7.Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond J. Mooney, Trevor Darrell, Kate Saenko. (2015). Sequence to Sequence -- Video to Text. , 4534–4542[10.1109/iccv.2015.515]
  8. 8.Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville. (2015). Describing Videos by Exploiting Temporal Structure. , 4507–4515[10.1109/iccv.2015.512]
  9. 9.Chia‐Wei Liu, Ryan Lowe, Iulian Vlad Serban, Mike Noseworthy, Laurent Charlin, Joëlle Pineau. (2016). How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. , 2122–2132[10.18653/v1/d16-1230]
  10. 10.Yang Gao, Oscar Beijbom, Ning Zhang, Trevor Darrell. (2016). Compact Bilinear Pooling. , 317–326[10.1109/cvpr.2016.41]
  11. 11.David Chen, William B. Dolan. (2011). Collecting Highly Parallel Data for Paraphrase Evaluation. , 190–200
  12. 12.Yingwei Pan, Ting Yao, Yehao Li, Tao Mei. (2020). X-Linear Attention Networks for Image Captioning. , 10968–10977[10.1109/cvpr42600.2020.01098]
  13. 13.Chiori Hori, Takaaki Hori, Teng‐Yok Lee, Ziming Zhang, Bret Harsham, John R. Hershey, Tim K. Marks, Kazuhiko Sumi. (2017). Attention-Based Multimodal Fusion for Video Description. , 4203–4212[10.1109/iccv.2017.450]
  14. 14.Shu Kong, Charless C. Fowlkes. (2017). Low-Rank Bilinear Pooling for Fine-Grained Classification. , 7025–7034[10.1109/cvpr.2017.743]
  15. 15.Bairui Wang, Lin Ma, Wei Zhang, Wei Liu. (2018). Reconstruction Network for Video Captioning. , 7622–7631[10.1109/cvpr.2018.00795]
  16. 16.Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, Ajmal Mian. (2019). Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning. , 12479–12488[10.1109/cvpr.2019.01277]
  17. 17.Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, Wei Liu. (2019). Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network. , 2641–2650[10.1109/iccv.2019.00273]
  18. 18.Shaoxiang Chen, Yu–Gang Jiang. (2019). Motion Guided Spatial Attention for Video Captioning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 8191–8198[10.1609/aaai.v33i01.33018191]
  19. 19.Sheng Liu, Zhou Ren, Junsong Yuan. (2020). SibNet: Sibling Convolutional Encoder for Video Captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9), 3259–3272[10.1109/tpami.2019.2940007]
  20. 20.Shizhe Chen, Jia Chen, Qin Jin, Alexander G. Hauptmann. (2017). Video Captioning with Guidance of Multimodal Latent Topics. , 1838–1846[10.1145/3123266.3123420]
  21. 21.Yaosi Hu, Zhenzhong Chen, Zheng-Jun Zha, Feng Wu. (2019). Hierarchical Global-Local Temporal Modeling for Video Captioning. , 774–783[10.1145/3343031.3351072]
  22. 22.Jingyi Hou, Xinxiao Wu, Xiaoxun Zhang, Yayun Qi, Yunde Jia, Jiebo Luo. (2020). Joint Commonsense and Relation Reasoning for Image and Video Captioning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07), 10973–10980[10.1609/aaai.v34i07.6731]
  23. 23.Wangli Hao, Zhaoxiang Zhang, He Guan. (2018). Integrating Both Visual and Audio Cues for Enhanced Video Caption. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1)[10.1609/aaai.v32i1.12330]
  24. 24.Yongqing Zhu, Shuqiang Jiang. (2019). Attention-based Densely Connected LSTM for Video Captioning. , 802–810[10.1145/3343031.3350932]
  25. 25.Xiangxi Shi, Jianfei Cai, Shafiq Joty, Jiuxiang Gu. (2019). Watch It Twice. , 818–826[10.1145/3343031.3351060]
  26. 26.Hui‐Yun Wang, Youjiang Xu, Yahong Han. (2018). Spotting and Aggregating Salient Regions for Video Captioning. , 1519–1526[10.1145/3240508.3240677]
  27. 27.Yunchen Pu, Martin Renqiang Min, Zhe Gan, Lawrence Carin. (2018). Adaptive Feature Abstraction for Translating Video to Text. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1)[10.1609/aaai.v32i1.12245]
  28. 28.Yitian Yuan, Lin Ma, Jingwen Wang, Wenwu Zhu. (2020). Controllable Video Captioning with an Exemplar Sentence. , 1085–1093[10.1145/3394171.3413908]
  29. 29.Junbo Wang, Wei Wang, Yan Huang, Liang Wang, Tieniu Tan. (2018). Hierarchical Memory Modelling for Video Captioning. , 63–71[10.1145/3240508.3240538]
  30. 30.Yiqing Huang, Qiuyu Cai, Siyu Xu, Jiansheng Chen. (2020). XlanV Model with Adaptively Multi-Modality Feature Fusing for Video Captioning. , 4600–4604[10.1145/3394171.3416290]
  31. 31.Karen Simonyan, Andrew Zisserman. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv (Cornell University)[10.48550/arxiv.1409.1556]
  32. 32.Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, Wojciech Zaremba. (2015). Sequence Level Training with Recurrent Neural Networks. arXiv (Cornell University)[10.48550/arxiv.1511.06732]
  33. 33.Jun‐Young Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio. (2015). A Recurrent Latent Variable Model for Sequential Data. arXiv (Cornell University)[10.48550/arxiv.1506.02216]
  34. 34.Ashwin K. Vijayakumar, Michael Cogswell, Ramprasath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, Dhruv Batra. (2016). Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence\n Models. arXiv (Cornell University)[10.48550/arxiv.1610.02424]

Cited By

  1. 1.Mingkang Tang, Zhanyu Wang, Zhaoyang Zeng, Xiu Li, Luping Zhou. (2022). Stay in Grid: Improving Video Captioning via Fully Grid-Level Representation. IEEE Transactions on Circuits and Systems for Video Technology, 33(7), 3319–3332[10.1109/tcsvt.2022.3232634]
  2. 2.Ping Li, Pan Zhang, Tao Wang, Huaxin Xiao. (2022). Time–frequency recurrent transformer with diversity constraint for dense video captioning. Information Processing & Management, 60(2), 103204[10.1016/j.ipm.2022.103204]
  3. 3.Norah Alrebdi, Amal A. Al-Shargabi. (2024). Bilingual video captioning model for enhanced video retrieval. Journal Of Big Data, 11(1)[10.1186/s40537-024-00878-w]
  4. 4.Tanzila Kehkashan, Abdullah Alsaeedi, Wael M. S. Yafooz, Nor Azman Ismail, Arafat Al-Dhaqm. (2024). Combinatorial Analysis of Deep Learning and Machine Learning Video Captioning Studies: A Systematic Literature Review. IEEE Access, 12, 35048–35080[10.1109/access.2024.3357980]
  5. 5.Zhu Liu, Teng Wang, Jinrui Zhang, Feng Zheng, Wenhao Jiang, Ke Lu. (2022). Show, Tell and Rephrase: Diverse Video Captioning via Two-Stage Progressive Training. IEEE Transactions on Multimedia, 25, 7894–7905[10.1109/tmm.2022.3232022]
  6. 6.Yifan Lu, Ziqi Zhang, Chunfeng Yuan, Peng Li, Yan Wang, Bing Li, Weiming Hu. (2024). Set Prediction Guided by Semantic Concepts for Diverse Video Captioning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(4), 3909–3917[10.1609/aaai.v38i4.28183]
  7. 7.Kishor Kumar Bhaumik, Fahim Faisal Niloy, Saif Mahmud, Simon S. Woo. (2024). STLGRU: Spatio-Temporal Lightweight Graph GRU for Traffic Flow Prediction. Lecture notes in computer science, 288–299[10.1007/978-981-97-2266-2_23]
  8. 8.Thomas Winterbottom, Sarah Xiao, Alistair McLean, Noura Al Moubayed. (2022). Bilinear pooling in video-QA: empirical challenges and motivational drift from neurological parallels. PeerJ Computer Science, 8, e974[10.7717/peerj-cs.974]
  9. 9.Pengjie Tang, Jiayu Zhang, Hanli Wang, Yunlan Tan, Yun Yi. (2025). SRVC-LA: Sparse regularization of visual context and latent attention based model for video description. Neurocomputing, 630, 129639[10.1016/j.neucom.2025.129639]
  10. 10.Pengjie Tang, Hong Rao, Ai Zhang, Yunlan Tan. (2024). Video emotional description with fact reinforcement and emotion awaking. Journal of Ambient Intelligence and Humanized Computing, 15(6), 2839–2852[10.1007/s12652-024-04779-x]
  11. 11.Pengjie Tang, Yunlan Tan, Jiewu Xia. (2023). Deep sequential collaborative cognition of vision and language based model for video description. Multimedia Tools and Applications, 82(23), 36207–36230[10.1007/s11042-023-14887-z]
  12. 12.K R Sarath Chandran, Dejah Madhusankar, Krithika Swaminathan. (2025). Real-time video captioning using feature fusion and attention mechanism. Signal Image and Video Processing, 19(12)[10.1007/s11760-025-04622-6]
  13. 13.R Guetari. (2022). Peer Review #2 of "Bilinear pooling in video-QA: empirical challenges and motivational drift from neurological parallels (v0.1)". [10.7287/peerj-cs.974v0.1/reviews/2]
  14. 14.(2022). Peer Review #1 of "Bilinear pooling in video-QA: empirical challenges and motivational drift from neurological parallels (v0.1)". [10.7287/peerj-cs.974v0.1/reviews/1]
← All publications