Go's whole approach to concurrency is about avoiding sharing memory; channels suspend in much the same way as async/await yield points.
Isn't that exactly what Go people push? "Don't communicate by sharing memory; share memory by communicating" and all that. If you start pushing pointers to shared memory around then I'd expect all of the problems of traditional multithreading to reappear.
Passing pointers to shared memory is the foundation of a huge number of idiomatic, performant, and productive design patterns and architectures. There exist a number of conventions and tools, like the race detector, which reduce the risks of data races to entirely reasonable levels.