"Sci-fi" already tells you what's the fictional part of the story: the science/tech, and maybe some fundamental rules around those. Fantasy is where you get to rewrite the basic rules of the universe.
In the span of a few decades we went from no planes to flying at almost Mach 10 in air (X-43) and 600.000km/h in space (Parker probe), and from AM radio to a super-computer in everyone's pocket. So you can bet any person can reasonably imagine that in a few centuries we could be advanced enough to build today's sci-fi. That's because they have a few real anchor points in the history of human civilization or even their own lifetime and an easy assessment of the progress of science between them.
Most people also actually think asteroid belts are that dense. But that's because even basic concepts about gravity aren't immediately apparent from day to day experience. Basic grasp of gravity will never allow you to believe it naturally "wears off" in an asteroid field so the chunks stay close enough to even collide but magically never clump together, or that 1000 years "in the future" will change what was constant for billions of years.
It only seems arbitrary if you don't bother to think about it.