What about Apple's services and 3rd party apps? I remember upgrading things like Notes with warning that they'll be unusable on old operating systems. I agree with you that for old devices like iPhone 4S, iOS 6 was the most usable iOS. I'm using iPhone 4S and it's a huge pain with iOS 9, so I'm thinking about trying to downgrade it, but I'm worried about ecosystem.
That said, I played with new iPhones at store and they are quite fast even with latest iOS. I guess I'll finally upgrade to 8, I hope that it'll last longer than 4S which became obsolete just a two years after release, when iOS 7 was released.