Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT) studies how vision-language models can collaboratively adapt prompts at test time when clients have private, heterogeneous data with no access to the original source data. We use CLIP, a vision-language model, to examine how existing test-time prompt tuning methods (TPT, SWAPPROMPT, DART, ADAPROMPT) interact with federated aggregation under IID and non-IID settings, and why naive prompt sharing can fail as client distributions diverge. To address this, we propose FedRAPP, a reliability-aware personalized aggregation framework, which uses label-free signals such as entropy reduction, prompt drift, and update similarity to determine which client updates should be combined. Across multiple CLIP and ImageNet benchmarks, we study how reliable aggregation can make source-free federated test-time adaptation more robust under distribution shift.
Team: Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry, Amin Ahsan Ali, AKM Mahbubur Rahman
