Those things shouldn't slow you down at all if you aren't using them: this is a browser optimization issue. The browser knows that the SVG has not changed in any way, and can with absolute accuracy discern that it can simple blit the pre-rendered raster.
Browsers have not optimized SVG at all, painfully clear when the simplest retained dynamic SVG graphic is magnitudes slower than building entirely new framebuffers from scratch for every frame, ala canvas. It is trivial to see that the former should see dramatic optimizations that make it a significant winner, but in actual practice that just hasn't happened.
So it's a classic chicken/egg thing. No browser maker does anything to optimize SVG because few use it. Few use it because it's so unoptimized.
The findings of this post remain the same in either case, and is completely correct, but it shouldn't be this way. It isn't right that this test favors PNG.