Ah, that's what I missed. So it's technically possible to have all of the niceties of GitHub without having to use GitHub?

Absolutely correct: GitHub just put a GUI on top on that, although it is a very neat one.