Turn it on, click the 'Buzz' label, scroll down to the bottom, and click "Turn Buzz off".
I mean, they opted-in how many millions of people? It isn't surprising that some percentage want it off, or that they consider gmail a tool for JUST email, or they don't want all that in the way.
I personally was wary of it, but I really do like how easy the implementation is, and already see comparable traffic in Buzz to what I do on my Facebook -- only I have less clicks to get to it.