OS X without Xcode installed doesn't have a compiler - if you can't compile, you can't use macports or homebrew.
It would be "neat" as in clean. No drama on Apple's behalf. Just put a price tag on the only real compiler, and it's over. Unless someone solves that issue.