I could not disagree more — allowing a third party to own the most crucial data in your app (the users) is a big mistake. I’ll admit to only having experience with Firebase for third-party auth, but in my experience the drawbacks of splitting auth and user related code between two systems soon outweighs the benefits of getting up and running quickly.
Auth code is undifferentiated, but if you have experience building apps it’s also not particularly difficult. I’d recommend either picking an open source solution that you host yourself or just taking a few days to grind it out.