Totally agree. Once you've got enough rooms to cover everyone's needs (kids don't have to share a room etc), pretty much any extra expansion is just more stuff to clean and maintain. Or if you're rich enough, more stuff to waste money on someone else cleaning and maintaining.
I probably have more of an epicurean outlook than most, but surplus stuff just feels like more stuff for me to worry about.
Plus it takes ages to even get around in really big houses and you're always shouting down the hallway to find where someone is.