Using this logic, nobody should be allowed to learn to drive because inexperienced drivers have a higher incident rate.
Of course, this is absurd. Instead, we slap on some caveats (time/passenger restrictions) and accept the risk with the knowledge the driver will improve with time.
This is exactly the same for self driving vehicles. The only way to make them safer for public roads is to drive them on public roads. So if we slap on some caveats, such as a reduced fleet, we can make them safer while keeping incidents down.