Just use MMU hardware to insert 2GB [0] no man's land below and above Wasm memory and only allow signed 32-bit indexing (-2^31 — 2^31-1).
This way the attacker can only read sandbox memory (and the useless 2 GB no man's land, mapped to pages full of zeroes or whatever).
When there's no speculation involved or the data is simply out of "speculative range", Spectre is toothless.
[0]: 2GB is just a basic example. More may be required if for example something like x86 SIB (Scale Index Base) is used for multiplying the index by 2, 4 or 8.