Furthermore, the 4GB limit does not just concern physical RAM, it also applies elsewhere, like memory mapped files.
In general, it is unlikely that you will see poorer performance from 64-bit code compared to its 32-bit counterpart.
>> Most of the memory used by programs are pointers
This is a huge generalization, may I add.
Yes, I will not argue that cache usage is less efficient, but I am sure you know that register access is faster than cache access, and we double the amount of registers in x64.
You may double the number of registers, but you are still shuffling tons of data between your caches and memory, and that is going to be your constraint for most programs. Losing half of the 4mb or 12mb of your L2 or L3 cache because you use 64 bit pointers has a much bigger impact than doubling the number of registers.