This is why Netflix started fast.com - because it draws data from the same distribution points as their video streaming apps it means you can't prioritise the speedtest without also doing so for the video traffic or (more likely) you can't de-prioritise the video traffic without also getting bad scores in that particular speedtest. From Netflix's point of view it is an answer to people contacting support with "my speedtest results are fine, the problem must be your servers" when they are experiencing video lag/drops and other such problems and the issue is due to ISP traffic shaping or the ISP simply not having enough backhaul bandwidth.
A more reliable test might be taking part in a busy public torrent: that way you are testing against arbitrary locations so your ISP can't be setting different shaping rules for them. Just remember to throttle upstream when testing downstream and vice-versa or saturation in the other direction will slow control packets that will in turn give you lower results for the one you are testing. This may fall into another trap though: unless you limit the number of active streams it may be an unrealistic test as more generally most processes use a small number of streams (or just a single one), and if you limit the number of streams too much you might get a lower result because each swarm member you connect to may be fairly saturated and sharing its bandwidth amongst many connections.