I never really saw the point of Xubuntu--I mean, 10 minutes in Ubuntu will give you the same thing, why have an whole separate distribution when the biggest change is the default desktop environment?
I agree, though, that this distinction is not clear enough by reading Xubuntu (or Kubuntu, Edubuntu, etc) websites.
Me, I usually just install a very basic Debian and pile crap on top as I go :)