Because if cars are being driven at a speed where they are actually useful, then basic physics dictates that it is not safe to mix cars and pedestrians. The car can only stop so quickly, and you must also factor in that both the pedestrians and drivers are humans and will make mistakes in gauging distance, speed, visibility, and will not always have ideal response times. The legal "right of way" cannot ignore reality. To avoid deaths it is far better to designate areas where cars have right of way and where pedestrians have right of way and clearly mark them.

I agree that the US infrastructure doesn't do enough to support pedestrian and bicycle transport. Expecting people cross only at intersections that are a half-mile apart is unrealistic. Expecting people to cross at a cross-walk that is 5 yards away is not. Crossing just beyond an intersection is the most dangerous possible place to cross due to inherently poor visibility, and all the competing demands for attention that drivers and pedestrians encounter around an intersection. This isn't a matter of rights or relative importance, its a matter of physics and human ability.

Wishing away all the cars is not practical solution, which is what your post essentially amounts to.