Well, xorg-server is 1 package that doesn't depend on much on its own. What you're describing sounds more like bad client dependencies pulling in chunks of a desktop environment.