It sounds like you had a process running over night that needed enough memory to force X (or whatever) to swap. Without the swap, that memory-hungry process would presumably have run less optimally (if it got memory allocation failures and adjusted its behavior to utilize less memory) or it would have simply died. It's not so clear that having processes die because they can't get memory is better than having something else run slowly, in general.
Edit: Possible caveat: Maybe whatever was running overnight did a lot of disk I/O and the OS decided to cache it at the expense of moving idle processes to swap (not sure if Linux does that or not).