From my understanding, it's not that we're moving away from parts of the universe, but that the distance between us is growing so fast that light sent from one part will find that after traveling toward us for some amount of time, the remaining distance to travel is actually more than than when it started.
One way for the distance between two objects to increase is indeed for those objects to literally be moving in opposite directions through space. But the expansion of the universe itself causes the distance between two otherwise-stationary points to nonetheless increase. Put differently, it's the cosmic yardstick that's shrinking, not the entities that must necessarily be moving.
(This is also why two points can be "moving apart" faster than light speed, the cosmic speed limit.)