Sure, but that's you. The very vast majority of django devs will find it suits their purpose. The VERY vast majority of Django projects are small and fast. Myself, I've done half a dozen projects, one spanning 20 kLoc, and the rest very small. I've never had a problem with Django auth.
To recommend it as something you should do when you make a Django app is bad advice, particularly for the beginners, who will just be confused by the extra steps.
I recommend anyone who isn't familiar with Django simply do the "My First App" tutorial. After that, the dev will find their own way, and I doubt it will be much like yours, or mine.