My 3GS is almost unusably slow on iOS5.
http://www.anandtech.com/show/4956/apple-ios-5-review/15
Basically, as fast or faster than 4.3. But mine is a development-only device with no music or other content, so maybe that's the difference.
There are already ongoing efforts to port the 4.0 SDK images to run on the Nexus One and the Nexus S, and I guarantee there will be even better options once the source code is released.
Use the freedom you paid $600 for to run ICS on your phone anyways.