contrib.auth can be made to do most of these if you still want the ability to play nice with other people's code who expect you to use auth. You'll have to create and manage User objects which it sounds like you're unwilling to do, but you can't have it both ways.
I think having it there is better than not. In most circumstances it saves me a lot of time. If I come across a circumstance that I can't use it then I'm just back to where I would be anyways and will have to write a lot more code and modify 3rd party apps.