Yeah, the security angle is a big deal. The long and short of it is that ad networks and publishers are both apathetic to what a gigantic dumpster fire ad-serving is.
Publishers are eager to blame ad networks for the terrible performance, while ad networks just can't give enough of a shit to work on it. Publishers also are eager to blame ad networks when they're caught serving scams and malware, while networks just don't care.
I'm a huge proponent of making publishers serve their own ads from their own domains, so they are forced to take ownership of both the issue of technical competency and abuse.