Fair Federated Learning with Biased Vision-Language Models

Huimin Zeng, Zhenrui Yue, Yang Zhang, Lanyu Shang, Dong Wang


Abstract
Existing literature that integrates CLIP into federated learning (FL) largely ignores the inherent group unfairness within CLIP and its ethical implications on FL applications. Furthermore, such CLIP bias may be amplified in FL, due to the unique issue of data heterogeneity across clients. However, in identity-sensitive FL applications, model fairness (i.e., group fairness) is imperative for model development. Therefore, this work explores a critical question ignored by the existing literature: how can we build a fair FL framework using biased pre-trained VLMs (e.g., CLIP)? To address this problem, we propose a fairness-aware adaptation framework tailored for VLM (e.g., CLIP) in the context of FL, named Fair Federated Deep Visiual Prompting or FF-DVP. As implied by its name, trains a fair FL model with fairness-aware deep visual prompting (DVP). Moreover, incorporates modality-fused classification heads to learn client-specific knowledge and fairness constraints. These modules explicitly addresses a unique bias in FL, namely the bias triggered by data heterogeneity. We show that can be readily extended to prevailing parameter-efficient fine-tuning methods (e.g., adapter or LoRA) for debiasing. To the best of our knowledge, is the first to leverage biased VLMs for building fair FL frameworks. Extensive results on human face attribute recognition (FAR) applications suggest that effectively improves model fairness and training convergence, outperforming state-of-the-art baselines.
Anthology ID:
2024.findings-acl.595
Volume:
Findings of the Association for Computational Linguistics: ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
10002–10017
Language:
URL:
https://aclanthology.org/2024.findings-acl.595
DOI:
10.18653/v1/2024.findings-acl.595
Bibkey:
Cite (ACL):
Huimin Zeng, Zhenrui Yue, Yang Zhang, Lanyu Shang, and Dong Wang. 2024. Fair Federated Learning with Biased Vision-Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10002–10017, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Fair Federated Learning with Biased Vision-Language Models (Zeng et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-acl.595.pdf