This seems somewhat related to https://arxiv.org/abs/2309.16588. Some Researchers found that Vision Transformers would repurpose low-informative background patches of images for internal computations. They gave them an explicit token for this to better results!