Even things like Tau Zero are using relativistic time dilation as the plot driver.
I agree, and sometimes the line is drawn between SF being "things that are theoretically possible" vs. Fantasy where things are impossible. But then you have things like Egan's Clockwork Trilogy, which is "what if the laws of physics actually worked a bit differently in this specific way" but which I assume anyone would consider SF. As opposed to Brandon Sanderson's books, which could be described in a similar way, but are usually categorized Fantasy.
At the end, it's mostly a marketing and feeling thing. As one of my favorite authors put it, the different between SF and Fantasy sometimes comes down to - are you putting a tree or a spaceship on the cover of your book?
What would you say is the reason for categorising works differently? Can you see differences there or do you also think it's mostly marketing?
An author once wrote an intro to some short story. The story is part of a much larger futuristic scifi universe in which people have developed telepathy and other things through genetic means. And the specific short story was the first one he wanted to publish, and it was about a specific planet in that world, in which the whole story is basically a telepath coming into town and interacting with the population.
And the publisher returned the note that this wasn't scifi, it was fantasy. Because of course he did - stripped of the broader futuristic setting that the story takes place in, it's just a story of a wizard coming into town. Never mind that there are solid science fiction explanations for the "magic" - you don't get that in the short story.