Upgrading probably does reduce performance slightly, but is it a noticeable difference; or is it something people say because of degraded batteries and go "hurr durr update bad"?
I think another issue is that apps do tend to get more sluggish, since they probably expect to be running on newer hardware. If you stick to an older ios version, if they're deemed incompatible, they won't update. But as soon as you're on the latest ios, you get the app updates, too, if you've left the default on.
Google maps, for example, is clearly more sluggish than before. Apple maps is practically unusable. Youtube crashes reliably (but works fine in the browser...)