You don't really avoid those things, though. In the process you end up with index-use-after-index-free, double-index-free, and accessing index-unallocated array entries.
These are the exact same bugs the borrow checker prevents in main memory, just hidden from the borrow checker by adding a layer of abstraction.
These are memory-unsafety bugs. You still get the same wrong answers caused by coding errors. Random junk values and behaviours, when dereferencing use-after-free invalid pointers that point to memory reused for a new object. It's just that pointers are now called indexes, and the borrow checker doesn't check these pointers.
It's like turning the borrow checker off for this set of objects. Those long-lived entities you mentioned act as a mechanism to enable that. That's useful to do, but nobody should be under the impression use-after-free, pointers to the wrong objects and other memory-unsafety bugs don't happen in the array index model.