This is pretty much spot on: when you go this route, a lot of people assume that you're unemployed involuntarily.
To them, working is a subset of having a job. They couldn't imagine themselves choosing to not have a job before retirement, and they assume that you look at it the same way.
The bottom line is that one of the hardest things in the world is looking at things from the other person's point of view. The truth is that it's something that all of us have to actively work on.