I think gmail adds some heuristics on top of it, like if the same image was included in emails to multiple people.
At least that's what I remember from them announcing the feature. No idea about other providers, and I haven't tested the feature myself.