ClipSwap: Towards High Fidelity Face Swapping via Attributes and CLIP-Informed Loss

This paper introduces ClipSwap, a new frame-work designed for high-fidelity face swapping. Earlier methods for face swapping often struggle in identity transfer due to the mismatches in attributes between the target and source images. To handle this issue, an attributes-aware face swapping approach is proposed in our work. We use a conditional Generative Adversarial Network and a CLIP-based encoder, which extracts rich semantic knowledge to achieve attributes-aware face swapping. Our framework uses CLIP embedding in the face swapping process for improving the transmission of source image's identity details to the swapped image by refining the high-level semantic attributes obtained from the source image. And source image serves as the input reference image for CLIP and ensures a more accurate and detailed identity representation in the final result. Additionally, we apply Contrastive Loss to guide the transformation of source facial attributes onto the swapped image from various viewpoints. We also introduce Attributes Preservation Loss, which penalizes the network to keep the facial attributes of the target image. Thorough quantitative and qualitative evaluations on multiple datasets illustrate the high-quality swapping results. Our proposed ClipSwap outperforms prior state-of-the-art (SOTA) methods in face swapping, particularly in terms of identity transfer and facial attribute features.