I've just turned off location accuracy, google maps displayed a popup said results will be better if I turn it on, and after dismissing the popup it did use GPS to localize me with no issues.
There are though some more privacy friendly alternatives that I remember from playing with GrapheneOS settings.
And on phones. In my experience, a hot start fix (with up to date ephemeris) shouldn't take more than a couple seconds. Maybe you have a bad antenna.