There is a trade-off between speculation and realism, that leaves a very, very narrow space for an interesting story.
Part of it is due to the fact that most of modern science has been mined mercilessly for "hard" sci-fi subjects, that have now become tropes that can't form the basis for an interesting new story anymore. For example, try to write a story where the entire premise is that someone manages to construct an Alcubierre drive. You'd get ...Star Trek: First Contact. Nice movie, hey. But nothing new, there.