I disagree (the rest of your comment I fully agree with, FYI). What I personally found difficult is to understand that: when you dereference an invalid memory address, then the dereferencing operation is working, but the OS/runtime environment won't allow you to do it!
Wait, what? Why?! Who cares my value is wonky, why is my program giving an error? Just obtain the wonky value! (is what I thought when I first did it)
The reason it's so difficult because at this stage you're still using printf for debugging, while you really should use a debugger, to really understand what's going on under the hood. Preferably, one also learns a modicum of x86 to really see it immediately.
The concept of dereferencing the wrong values isn't explained a lot. How to troubleshoot segfaults isn't explained at all in beginner tutorials, not even in my own (I just wrote an explanation for fun as a comment in the same thread [1]).
Also FWIW, when I heard the concept of pointers I found them intuitive immediately. I tend to think and reflect a lot, and the explanation of referencing and dereferencing almost sound like some proverbial life wisdom to me.
I found their use a lot harder, in part because I found code syntax hard. In another part, because I used to find "doing things" and "building things" (of any kind) an insurmountable task.
---
With that said, I do agree with the rest of your explanation. I suppose it also depends on who teaches pointers to you. I got a classroom explanation first, a few examples and then some exercises. Then suddenly, I had to write a 500 line C program. Then all the complexity storms at you like a rabid zombie army.
But if you're used to writing small C programs all the time and picked up on pointers that way, but then suddenly have to explain it. Then, I can see how that's hard.