The Huntington Estate is definitely worth a day (and an excellent place to take the family) if you want to avoid the usual tourist traps in Southern California. Gardens, art, great food, not too crowded even.
I'm amazed at how many of us in Los Angeles live here for years and skip this place. It's one of the most beautiful places I've ever been.