The extra cost should be born by society, which is where the requirement comes from. If we think the disabled should ride for the same price we should make that part of medical coverage and pay the driver for the extra time spent.
As is, any sensible driver will ditch these fares with any means possible. More work, less money, and generally bitchier customers. The next time your fare comes up they'll pull over and take a bathroom break rather than responding.
If we want something, we have to pay for it, not try to stick someone else with the bill.