(Former) Flash nerd here. The problems with plackback and scrubbing are absolutely the fault of the Flash player and there's really nothing you can do about it short of hacking your own video handling into the SWF format. There are a lot of low-level network-related problems in Flash.

I have zero real proof, but I know for a long time Adobe (Macromedia) was pushing their streaming server and didn't really put a lot of effort into handling streaming/buffering without it and I suspect this just never got fixed as things were "good enough".