Eventually, that’s likely to be true, but currently these are sill in the experimental stage and regularly have issues. The collective benifit is probably a net win long term. But, they are not replacing google street view cars or something so as to spread that risk out. Instead they are concentrating that risk into a small subset of residential streets in some poor neighborhoods.
Even if it was already slightly safer than human drivers the extra traffic really risks these people’s lives. But because they are low income they don’t expect to be able to afford self driving cars in the next 10+ years. So, that’s meaningful harm without the exception to benifit.
Now how a tiny subset of a community responds is predictable but not justifiable.