You are absolutely right, but do check out Greg Egan's "Diaspora" for a hard scifi that tries to get around that constraint.

More seriously, and slightly less scifi, you can have more than 3 spacial dimensions and still have stable gravity/electromagnetism if the topology of these dimensions is a bit weirder (very small periodic dimensions for instance).

Why couldn't the 4th physical dimension interact with our 3 spatial dimensions as something we have trouble explaining such as quanta or gravity? There is a lot our physical bodies are not even aware due of to our senses such as most of the electromagnetic spectrum and gravity.