If you think space is something that can be littered you might be conflating space with Low Earth Orbit.
The space in the solar system alone is literally quadrillions of times the volume of the entire Earth.
The space in the solar system alone is literally quadrillions of times the volume of the entire Earth.
Each day we wait to explore the universe, millions of planets effectively cease to exist.
I've not made up my mind about this whole expanding universe thing (whether it's permanent, local, even real) but based on our current understanding it certainly seems that you're right - the world is getting a little bit smaller all the time. Time is of the essence - we need to move fast.