Nothing stayed where the big bang was, but everything flew away at the same speed. There is a flaw in your imagination, because as a being in 3D you probably cannot imagine how a 3D thing explodes (I certainly can't). Don't think of the universe as sphere-shaped -- it is not a 3D sphere. Think of it as the 3D surface of a 4D sphere.

Why? Let's reduce by one dimension to be able to imagine this: imagine a small thing (a grain of sand) explodes in our 3D world and all the tiny pieces are flying away at equal speed, then the structure that is created by the explosion is not a 3D sphere, but the surface of a sphere, a 2D structure, because where the explosion started, there is nothing left of the original thing, as every particle of it has moved away from that point, so at any point in time after the start of the explosion, the particles are on the surface of a sphere. That surface is a 2D structure embedded into our 3D world, much like the surface of an expanding balloon. In the same way, our universe is a 3D structure, so looked at from the outside (which we obviously cannot do), it is not a 3D sphere, but probably more like the surface of a 4D sphere.

Now imagine us as a dot on a balloon that is expanding, and other stars are also dots on the surface of the balloon. In whichever direction you look, all the stars are moving away from you (the balloon is expanding). You do not need a center of this structure to observe this moving away -- nothing is in the center -- there is no center.