On rail I'm not as sure but on bus yes. Drivers are the largest cost associated with a bus line. There's also a whole set of downstream costs like bathroom breaks which requires that routes are aligned with bathroom stops and that bathrooms are kept in good working order. Breaks also decrease bus frequency (humans need breaks!) and running more buses is often limited by the number of drivers you can hire.
However bus drivers often play a dual role in US transit of discouraging anti-social behavior so it's unclear to me if you could even get rid of the bus driver and the associated inefficiencies or you'd just need to replace them with a police officer and deal with the exact same problems.
Many bus drivers are unhappy having to play this role, so that's also a factor.