Correct. Multi-point racing harnesses can only be used with a cage. The standard diagonal shoulder belt design allows your body to lean over sideways (towards the center of the car) if the roof of the vehicle caves in on your head and pushes you down. If you're in a harness without a rollcage and the roof caves in, the harness locks your spine upright and you end up with a head/neck/spine injury that's very likely to be fatal. This whole line of reasoning snowballs from there, because beyond that:
If you do install a well-designed cage just so you can safely use a harness, now you also need to wear a helmet. A cage without a helmet is begging for a bad head injury, because your head bounces around a little during an accident, and those bars are hefty, rigid, immobile objects mere inches away from your skull.
Even in a fairly low-speed incident that wouldn't cause other injuries, the bars would put a dent in your naked skull. You're also going to want a neck restraint system for that helmet (it's like a collar under the harness with little straps to the helmet, so that it can't move far), in part because of the added head weight (momentum) from the helmet and how rigidly your torso is locked in place, to prevent neck injuries.
Now that you have all those parts down: the combination of the harness locking down your torso, the neck restraint, and the limited window of visibility from the helmet means you've lost a good chunk of your peripheral vision and can't turn your head/neck to check things either. With your vision more or less locked in straight forward, you can't see all the things you really need to see to drive in normal street conditions safely. The limited vision works fine on a racetrack under racing conditions because everyone's trained the same way and operating in a sanitized environment, with no "intersections" or cross traffic, you have flag workers and/or radios to warn you of some things you can't see, etc.
The bottom line is that both street and race car safety systems are a whole complex of things that are engineered together in concert. They're well-tuned to the appropriate environment, and foolishly mixing ideas from the two generally makes you less safe, not more.
Random context video showing a race crash from the inside and outside perspective. Loss of brakes at the end of a high speed straight at COTA (he was still moving at 136 mph at the wall impact, after all the attempts to slow it down). The driver walked away fine, thanks to all the safety gear that the cockpit video shows off nicely near the end: https://www.youtube.com/watch?v=dQitOyEyRd0 .