Or is it really? What if it's staying perfectly still, and YOU'RE the one moving?
... How could you even tell?
The principle of Relativity is that you cannot tell any difference, for the laws of physics are the same whether you are moving or the galaxy is moving. The theory of Special Relativity describes which mathematical transformations you need to go from one description to the other (in the absence of gravity, which is a General Relativity concern). Philosophically, the way of unifying these involves mixing up Space and Time, in which various observers cannot agree on a single version of things like speeds and time and energy levels, into a unified Spacetime, which they all can agree on.
"Changing your reference frame" simply means picking something different that you consider to be at rest.