Xiaotian Yu

Not All Attention Heads Are What You Need: Refining CLIP's Image Representation with Attention Ablation

2025 · arXiv (Cornell University)

This paper investigates the role of attention heads in CLIP's image encoder. Building on interpretability studies, we conduct an exhaustive analysis and find that certain heads, distributed across layers, are detrimental to the resulting representations. …