SDE 2 is a career position at Amazon — and it sounds like that’s what you want: to stay at that level, but working as the “point man” on increasingly important technical systems.
I would say that SDE 3 (senior) and SDE 4 (principal) are fundamentally political — you’re responsible for aligning the technical vision between teams and across whole organizations, as well as training SDE 1s and SDE 2s. You’re going to need to communicate, a lot, to do that.
Those roles aren’t management, but you’re also not an IC at SDE 3/4 the same way you’re an IC at SDE 1/2.
I think it’s okay to turn down promotions you don’t want.