I am on Firefox 5 and I don't see visible differences between them anymore. Yes, Firefox 3.x used to be visibly slower.
I also don't see visible differences in boot time, but this depends a lot on how many extensions you have -- I only use Firebug and the Web Developer toolbar. And at least Firefox has plugins that aren't totally useless.