I'm also relatively fluent and I'm the same way as you. You need to be Cervantes-level fluent, like my wife who was not only native born in a Spanish speaking country but educated in private schools and then college educated there as well. I also think some people may naturally be slightly more in tune with linguistics in general.

Spanish speakers are a huge group and many simply don't know their own language. It's like going to Alabama and expecting to find Shakespeare.

I'll grant you that mutual intelligibility is difficult to define but I'm using it in the sense that my wife is conversational with folks from Portugal and Italy when we travel there. That's a decent standard.