Not necessarily. The gif is requested once and cached, and then rendered once and cached.
The CSS triangle would get rendered over and over again.
The CSS triangle would get rendered over and over again.
(If I have misunderstood some step of the rendering process, I would love to know how, or what. I don't know much about graphics processing, so I am unaware of any caches beyond what the browser maintains.)
With SVG, the browser will render it once and keep a bitmap, then composite it.
With a CSS icon, there is no way for the browser to know that it's an icon at all. There is no clear delimitation between it and the rest of the page. The browser has no choice but to re-render it every single time.
Images or SVG are much better choices for icons than a bunch of CSS.