Rolling the clock backwards toward the big bang, it is not matter which is directly compressed, it is the fabric and metric of space itself. Matter being compressed is merely a side effect of the fact that "all of the locations that the matter occupies" are also compressed (though again, remain aware that calling it "compression" only makes sense when you rewind time in our cosmological model backwards).

So it is not as though you and the Andromeda Galaxy are made out of matter that got flung out of a point explosion long ago so that now you have traveled a very long distance away from one another, it is more like "both you and the Andromeda galaxy sat still for 13.8 billion years but space expanded between you in that time, so originally you were right on top of each other along with everything else".

We can rewind the model until the entire observable universe was as small as a Planck volume, but we have abundant evidence that the universe is indefinitely larger than that so even "that time when our 98gly diameter patch of space was almost indistinguishable from a mathematical point" means little when even that "point" was still just one pinprick out of the smooth manifold of a larger universe which could have been stupidly large or infinite even that early on.