Do you have a lot of visitors? I have my webmaster tools interface in Swedish, but it is telling me that my 1,5 sec (faster than 84% of all sites) average is of low accuracy ("låg tillförlitlighet") and has less than 100 data points.
The worst offenders to the speed of my site is that I have both google adsense and google analytics on it. I find it ironic.
From what I understand the time is sampled from visitors, and includes the time from the page request until the page is fully rendered.
I'd love to know what people do that get their times well below sub-second.