Only iPhone 4S and iPhone 5 will be upgraded, right? Those have pretty good GPU's. It was iPhone 4 that had a pretty terrible one combined with the retina display at the time.
The biggest impact would be seen on iPads, because iPad 3 actually took a step backwards in terms in performance behind iPad 2, when it switched to retina, and iPad 4 barely matches iPad 2 in performance (if we're comparing native iPad 2 apps vs native "retina" iPad 4 apps).
It could also be the reason why the next iPad is rumored to not have a huge battery anymore, like iPad 3 and iPad 4, and be much slimmer (the last 2 iPads were thicker than iPad 2).