Specifics: web browsers (and specifically Firefox, but it's true for WebKit-based browsers too) tend to be slower on most web workloads when running 64-bit.
There are several reasons for this.
First of all, web browsers are very pointer-chasing heavy because of the nature of web specs: the DOM is all about tree traversals, and so is CSS layout. Further, you have to share style, font, etc data across objects to get anything resembling sane memory usage, which means chasing pointers to all that data. Given that many of the algorithms involved require traversing large chunks of the DOM tree and various ancillary data structures, the upshot is that most of the really processing-intensive parts of a browser layout engine (possibly excluding graphics) are in fact more cache-bound than register-starved.
The second reason is a bit of a chicken-and-egg problem. Since most of the users are running the 32-bit builds, the 32-bit JIT has had more optimization work on it than the 64-bit one. The obvious solution is to improve the 64-bit JIT, of course.
The third reason is that even in JITTed code it turns out that cache locality matters a lot because JS objects share lots of their state across multiple objects so getting to it involves pointer-chasing. An actual JSObject in SpiderMonkey is 4 pointers and all the other things you want out of an object you reach through those 4 pointers. This reduces memory usage enormously compared to putting the data inline in the objects, and actually improves cache locality in general, but exacerbates the difference between 32-bit and 64-bit.
You're right that for heavily numeric JITted code x86-64 beats the pants off x86-32. Unfortunately most JS out there is object-heavy pointer-chasing, not numeric code...