I heard about this, and avoided pointers for a long time because of a fear that comments like yours gave me.
I then ended up realizing Java objects that I had been dealing with were just memory references, and that led me to learning about pointers in C.
Once I realized that I basically already knew what pointers were, I was pretty confused about why it created so much difficulty. Like it was an actual barrier for me to really understand how simple they are, so few moving parts.
It seems to me that everyone intuitively understands the difference between a home address, and the home itself.
Not to say that I'm an expert in pointer arithmetic or anything, I still prefer symbolic programming to dealing with pointers directly, for the same reason I prefer spoken English to Morse code.
I definitely agree that stuff is pretty annoying.
I guess I've gotten used to it to the casting behavior. Though these days I program mostly in dynamically typed languages.