Really ? With an SSD you can even go with 2GB of RAM you won't see any difference.
My point is, with an SSD, you can swap, it won't be slow as the good old HDD days. On my workstation with 8GB of RAM and a 512 SSD (MBPr), I often have 8 or more GB swapped, it's not noticeable at all.
The only downside is that yeah, the VM file size will reduce your actual available disk space until you clean it up (reboot).