Most likely reason is to avoid drivers being robbed - if there's no (easy and accessible for the driver) way to take money out of the thing, local addicts won't get an idea "let's jump on a bus and take money from the driver" because driver can't really give the money.

Also a bit of avoiding fraud from driver's side - if the driver is allowed to take money out, it's impossible to control whether all the payments actually got to the end point.

That said, I've seen systems where the drivers give change - e.g. in Israel - and that's wasn't a big deal. So I guess it also depends on the locality.