Conference Paper2026

R-MMA: Enhancing Vision-Language Models with Recurrent Adapters for Few-Shot and Cross-Domain Generalization

Md Fahim, Md Farhan Ishmam, Mir Sazzat Hossain, M Ashraful Amin, Amin Ahsan Ali, AKM Rahman

Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

In: 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

IEEE, pp. 6496-6506

DOI: 10.1109/wacv61042.2026.00628URL

Abstract

Pre-trained vision-language models (VLMs) such as CLIP exhibit strong generalization but struggle with few-shot adaptation due to the trade-off between gaining task-specific knowledge and preserving general performance. While multimodal adapters add trainable modules that improve alignment and excel in few-shot generality, they greatly increase the trainable parameter count while relying heavily on the prior layer’s frozen representation. Addressing these limitations, we introduce Recurrent Multi-Modal Adapter (R-MMA), a lightweight and efficient adapter that uses self-attention to compute a unified latent representation with a single set of shared adapter weights. Our attention-based alignment harmonizes the adapter outputs with the frozen encoder features before fusing the modalities, ensuring better preservation of pre-trained representations and cross-modal consistency. Our experiments show that R-MMA achieves state-of-the-art performance on most datasets for base-to-novel generalization, cross-dataset evaluation, and domain generalization, under few-shot settings. Our approach also achieves one of the highest forms of parameter efficiency with only a few trainable weight matrices for the whole network, regardless of its depth. Our code is available at: https://github.com/farhanishmam/R-MMA.

Keywords

  • vision-language models
  • domain generalization
  • cross-dataset generalization
  • self-attention
  • few-shot learning
  • CLIP
  • parameter efficiency
  • cross-modal consistency

CCDS Authors

MD Fahim
Md Fahim

MD Fahim

  • Research Assistant, CCDS

Published at

ACL, EMNLP, ECAI, WACV, LREC, ICPR, BEA, BLP, Information Sciences, Smart Agricultural Technology

Research Interests

Machine Learning, Natural Language Processing

Mir Sazzat Hossain
Mir Sazzat Hossain

Mir Sazzat Hossain

  • Research Assistant, CCDS

Bachelors

Independent University, Bangladesh

Published at

WACV, IJCNN, ICIP, Smart Agricultural Technology

Research Interests

Machine Learning, Computer Vision

Prof. M Ashraful Amin, PhD
M Ashraful Amin, PhD

Prof. M Ashraful Amin, PhD

  • Founder Director, CCDS
  • Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Published at

ACL, CHI, UbiComp, ECAI, WACV, PAKDD, IJCNN, ICPR, ICIP, HCII, CLEF, BLP, Scientific Reports, JMIR, IEEE Access, Smart Agricultural Technology, Frontiers in Computational Neuroscience, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

Amin Ahsan Ali, PhD
Amin Ahsan Ali, PhD

Amin Ahsan Ali, PhD

  • Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

ACL, CHI, UbiComp, ECCV, IJCAI, ECAI, WACV, IPSN, PAKDD, LREC, IJCNN, ICPR, ICIP, BEA, BLP, ICMLA, Scientific Reports, Scientific Data, Pattern Recognition, Information Sciences, Ad Hoc Networks, Drug and Alcohol Dependence, PLOS ONE

Research Interests

Artificial Intelligence, Machine Learning

References

  1. 1.Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. (2016). Deep Residual Learning for Image Recognition. , 770–778[10.1109/cvpr.2016.90]
  2. 2.Jakubův, Jan, Chvalovský, Karel, Goertzel, Zarathustra, Kaliszyk, Cezary, Olšák, Mirek, Piotrowski, Bartosz, Schulz, Stephan, Suda, Martin, Urban, Josef. (2023). Exploiting Generative AI to Scale up Intelligent Tutoring Systems. DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)[10.4230/lipics.itp.2023.19]
  3. 3.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei. (2009). ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255[10.1109/cvpr.2009.5206848]
  4. 4.Jonathan Krause, Michael Stark, Jia Deng, Li Fei-Fei. (2013). 3D Object Representations for Fine-Grained Categorization. , 554–561[10.1109/iccvw.2013.77]
  5. 5.Maria-Elena Nilsback, Andrew Zisserman. (2008). Automated Flower Classification over a Large Number of Classes. , 722–729[10.1109/icvgip.2008.47]
  6. 6.Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, Antonio Torralba. (2010). SUN database: Large-scale scene recognition from abbey to zoo. , 3485–3492[10.1109/cvpr.2010.5539970]
  7. 7.Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu. (2022). Learning to Prompt for Vision-Language Models. International Journal of Computer Vision, 130(9), 2337–2348[10.1007/s11263-022-01653-1]
  8. 8.Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, Andrea Vedaldi. (2014). Describing Textures in the Wild. , 3606–3613[10.1109/cvpr.2014.461]
  9. 9.Hao Tan, Mohit Bansal. (2019). LXMERT: Learning Cross-Modality Encoder Representations from Transformers. , 5099–5110[10.18653/v1/d19-1514]
  10. 10.Lukas Bossard, Matthieu Guillaumin, Luc Van Gool. (2014). Food-101 – Mining Discriminative Components with Random Forests. Lecture notes in computer science, 446–461[10.1007/978-3-319-10599-4_29]
  11. 11.Patrick Helber, Benjamin Bischke, Andreas Dengel, Damian Borth. (2019). EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7), 2217–2226[10.1109/jstars.2019.2918242]
  12. 12.Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu. (2022). Conditional Prompt Learning for Vision-Language Models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 16795–16804[10.1109/cvpr52688.2022.01631]
  13. 13.O. M. Parkhi, A. Vedaldi, A. Zisserman, C. V. Jawahar. (2012). Cats and dogs. , 3498–3505[10.1109/cvpr.2012.6248092]
  14. 14.Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, Justin Gilmer. (2021). The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 8320–8329[10.1109/iccv48922.2021.00823]
  15. 15.Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, Yu Qiao. (2023). CLIP-Adapter: Better Vision-Language Models with Feature Adapters. International Journal of Computer Vision, 132(2), 581–595[10.1007/s11263-023-01891-x]
  16. 16.Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer. (2023). Sigmoid Loss for Language Image Pre-Training. , 11941–11952[10.1109/iccv51070.2023.01100]
  17. 17.MD. Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, Hamid Laga. (2019). A Comprehensive Survey of Deep Learning for Image Captioning. ACM Computing Surveys, 51(6), 1–36[10.1145/3295748]
  18. 18.Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, Fahad Shahbaz Khan. (2023). MaPLe: Multi-modal Prompt Learning. , 19113–19122[10.1109/cvpr52729.2023.01832]
  19. 19.Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, Dawn Song. (2021). Natural Adversarial Examples. , 15257–15266[10.1109/cvpr46437.2021.01501]
  20. 20.Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li. (2022). Tip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification. Lecture notes in computer science, 493–510[10.1007/978-3-031-19833-5_29]
  21. 21.Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, Xinmei Tian. (2022). Prompt Distribution Learning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5196–5205[10.1109/cvpr52688.2022.00514]
  22. 22.Hantao Yao, Rui Zhang, Changsheng Xu. (2023). Visual-Language Prompt Tuning with Knowledge-Guided Context Optimization. , 6757–6767[10.1109/cvpr52729.2023.00653]
  23. 23.Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan. (2023). Self-regulating Prompts: Foundational Model Adaptation without Forgetting. , 15144–15154[10.1109/iccv51070.2023.01394]
  24. 24.Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo. (2022). AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition. , 16664–16678[10.52202/068431-1212]
  25. 25.Md. Farhan Ishmam, Md. Sakib Hossain Shovon, M.F. Mridha, Nilanjan Dey. (2024). From image to language: A critical analysis of Visual Question Answering (VQA) approaches, challenges, and opportunities. Information Fusion, 106, 102270[10.1016/j.inffus.2024.102270]
  26. 26.Lingxiao Yang, Ru-Yuan Zhang, Yanchen Wang, Xiaohua Xie. (2024). MMA: Multi-Modal Adapter for Vision-Language Models. , 23826–23837[10.1109/cvpr52733.2024.02249]
  27. 27.Hantao Yao, Rui Zhang, Changsheng Xu. (2024). TCP: Textual-Based Class-Aware Prompt Tuning for Visual-Language Model. , 23438–23448[10.1109/cvpr52733.2024.02212]
  28. 28.Sun-Ao Liu, Yiheng Zhang, Zhaofan Qiu, Hongtao Xie, Yongdong Zhang, Ting Yao. (2023). Learning Orthogonal Prototypes for Generalized Few-Shot Semantic Segmentation. , 11319–11328[10.1109/cvpr52729.2023.01089]
  29. 29.Dongjun Lee, Seokwon Song, Jihee Suh, Joonmyeong Choi, Sanghyeok Lee, Hyunwoo J. Kim. (2023). Read-only Prompt Optimization for Vision-Language Few-shot Learning. , 1401–1411[10.1109/iccv51070.2023.00135]
  30. 30.Cairong Zhao, Yubin Wang, Xinyang Jiang, Yifei Shen, Kaitao Song, Dongsheng Li, Duoqian Miao. (2024). Learning Domain Invariant Prompt for Vision-Language Models. IEEE Transactions on Image Processing, 33, 1348–1360[10.1109/tip.2024.3362062]
  31. 31.Adrian Bulat, Georgios Tzimiropoulos. (2023). LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models. , 23232–23241[10.1109/cvpr52729.2023.02225]
  32. 32.Hao Chen, Ran Tao, Han Zhang, Yidong Wang, Xiang Li, Wei Ye, Jindong Wang, Guosheng Hu, Marios Savvides. (2024). Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets. , 1551–1561[10.1109/cvprw63382.2024.00162]
  33. 33.Chen Xu, Yuhan Zhu, Haocheng Shen, Boheng Chen, Yixuan Liao, Xiaoxin Chen, Limin Wang. (2024). Progressive Visual Prompt Learning with Contrastive Feature Re-formation. International Journal of Computer Vision, 133(2), 511–526[10.1007/s11263-024-02172-x]
  34. 34.Yuncheng Guo, Xiaodong Gu. (2025). MMRL: Multi-Modal Representation Learning for Vision-Language Models. , 25015–25025[10.1109/cvpr52734.2025.02329]
  35. 35.Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, Changsheng Xu. (2025). Toward Visual Grounding: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3), 2749–2771[10.1109/tpami.2025.3630635]
  36. 36.Ting Liu, Xuyang Liu, Siteng Huang, Honggang Chen, Quanjun Yin, Long Qin, Donglin Wang, Yue Hu. (2024). DARA: Domain- and Relation-Aware Adapters Make Parameter-Efficient Tuning for Visual Grounding. , 1–6[10.1109/icme57554.2024.10688132]
  37. 37.Wei Dong, Dawei Yan, Zhijun Lin, Peng Wang. (2023). Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing. , 52548–52567[10.52202/075280-2288]
  38. 38.Thomas Rolland, Alberto Abad. (2025). Exploring Shared-Weight Mechanisms in Transformer and Conformer Architectures for Automatic Speech Recognition. , 2885–2889[10.21437/interspeech.2025-2733]
  39. 39.Li Fei-Fei, R. Fergus, P. Perona. (2005). Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. , 178[10.1109/cvpr.2004.383]
← All publications