How does single ownership conflict with the idea that memory is a flat array of bytes?

Further, the borrow checker does not care about pointers, only references. With pointers, you are on your own. It is true that using pointers in Rust is more cumbersome than it could be. But it is much easier to compartmentalise the pointer parts into separate functions and expose references instead.

I agree that some paradigms and patterns are genuinely difficult to use, e.g. any intrusive data structure, but I do not see the contentious link between simple memory models and the borrow checker and the like.