Now manage users and granular user-level access to repos... it's doable but you're going to write a boatload of one-off scripts and hacks, which tools like soft-serve have already implemented.
The point is that a git server isn't just ssh and git except in the most barebones simple use case.