I think you're right. I should've said x86 (or maybe IA-32?), not amd64.
clang, as far back as Compiler Explorer goes (i.e. clang 3.0.0, i.e. 2011), always assumes SSE for -m32; presumably because there's nothing to be backwards-compatible to, unlike gcc.
Doesn't look particularly good for "default will just change at some point", though we can hope.