Mythical Man Month[1]
Peopleware[2]
Very highly recommend you read both. The MMM is amazing because it's held true for over 35 years now.
[1]http://en.wikipedia.org/wiki/The_Mythical_Man-Month
[2]http://en.wikipedia.org/wiki/Peopleware:_Productive_Projects...)
Can't remember where I read that - might have been the MMM.
3 in unstable as too often 1 person feels left out, 4 is an even number so risk of split vote.
http://www.lifewithalacrity.com/2004/03/the_dunbar_numb.html
I'd agree that a larger team effectively mitigates risks around individuals not fitting in or decision-making being stymied, but in my own personal experience I've definitely preferred, due to a perception of maximising by own personal productivity, that 3 or 4 was best. That is, of course, completely subjective!
However, here is a more recent article that suggests that "best" team size is 5 plus or minus 2 - which fits into my preferred range:
In a given large system, you can and should have many small teams working independently.
Another small team should act as the 'curator' or 'middleman' for communication between teams.