I agree about the size. Especially for a sailboat, I would say smaller is better. I have more fun sailing a 12' Laser dingy than a 40' sailboat, simply because it is more responsive and you feel like you have more control.
And for sail and power boats alike, I don't think I would want to own a boat that I couldn't pilot myself or fit into most harbors. Otherwise it's basically a glorified floating luxury hotel and expensive status symbol.