The onslaught of open relay probing and inbound spam was unbelievable. At some points, I honestly thought I was under a targeted DDoS attack because there were so insanely many inbound connections for hours at a time. Even though I was always able to get it under control, new email-related fires were constantly popping up. Worse, handling all that traffic put considerable load on my server; capacity which would have been better allocated toward serving web traffic.
In the aftermath, I've also noticed that email I send through Google's SMTP server is less often flagged as spam on the receiving end, whereas I had trouble with mail sent through my self-hosted SMTP (and my own ISP's SMTP) ending up in the junk/spam folders at a lot of destinations. That was one of the more insidious drawbacks because it took me a while to fully realize.
At the scale where Google Apps is free (i.e. few enough users that you don't likely have a full-time sysadmin), it's difficult to understand not taking advantage of it. I value my time too much to waste it on unnecessary server administration.