My family and I built a two story house as a family cabin in 1992-4, which was a cool experience, albeit exhausting and humbling. The only things we didn't do were the foundation and some of the finish drywall. Everything else we did mostly in a single summer, from the first floor joists to the roof (which we did in 90F weather!). Great experience, really helped me understand what is behind the walls of my future homes.

I'd say my son, but really my wife is the one that built him ;)