You're right, I hadn't realized that the spec forbade primitives as WeakMap keys - so while to approach should actually work in other languages, in JS it would leak memory as you'd be forced to use a normal map (or clean up after yourself - ich).
Yeah, weak map have weak references to the keys, not the values.