The term refers to people who don't try to get into the art world in-group but still create interesting things, so "outsider" is an accurate and literal description of their status. However the in-group is uncomfortable with consciously engaging with this dynamic and the dysfunction it points to, so engages in a little double-think and invents more palatable justifications for using the term "outsider".
The idea of ignoring in-group games of meaning and status, instead creating works entirely from playful self-expression, is so alien it is best left outside.