I get the same, and it's not dependent on ISP. Many (most?) flash players are buggy resource hogs, and the html5 implementations I've seen don't feel ready for primetime yet.
I have zero real proof, but I know for a long time Adobe (Macromedia) was pushing their streaming server and didn't really put a lot of effort into handling streaming/buffering without it and I suspect this just never got fixed as things were "good enough".
If you're talking about streaming .flvs off of a regular Apache setup, then it shouldn't be surprising that scrubbing and seeking are troublesome: static file servers weren't build for that use case.
What would you have preferred Adobe to have done?