This is where you lose me:
"...I'd figure that if they were injecting JS to steal my password, or subtly poison my RNG, and if they were doing that to everyone..."
To everyone — maybe that would get found out, maybe it wouldn't. But it doesn't have to be done to everyone. That backdoor in the JS could be served to only you. Maybe only certain times of day. How are you going to know?