It seems to me that in a semantic embedding, "big", "huge", "enormous", and "gargantuan" should roughly point in the same direction but have different magnitudes. For instance, I might assume that the nearest neighbor to "big * 10" is "huge" or "enormous". But if embeddings must be normalized, I either can't tell the difference between these four terms, or they must point in different directions. I can't even talk about "big * 10" at all. I mean, perhaps there are "emphasis" dimensions roughly corresponding to words like "very" and "extremely", but it still seems like "big" and "enormous" would be hard to distinguish without scalars. It seems to me that the only significant difference between "big" and "huge" is in fact their magnitude, and forcing "huge" to point away from "big" so that it can be non-zero in the "very" dimension must cause it to be less "big-like" in some other small way, right?
I'm surprised that throwing away all the power of scalars is worth it. I'm not a professional in the field, so maybe there's good reason for this that I just haven't read. Can any professional comment on why vectors of magnitude != 1 are not helpful in embeddings?