Architectures/ABIs with 64bit pointers have a much lower probability of arbitrary values mapping to active memory space for the process, so memory bloat caused by the conservative GC strategy is much more limited.
The x32 abi looks like it would suffer from this 32bit conservative garbage collection problem too. Short of changing the VM strategy, using 64 bit pointers is the only way to ensure that such VMs do not accumulate so much unused but not-garbage-collectable memory allocations.
Performance. What other reason do you need? (Using cache more efficiently makes programs run faster).
(i can dig out a post of (afair) solar designer that goes into more detail, but i think this should be clear as is)