There is a toggle to control whether Google keeps track of your search history. I suppose there's a separate toggle for location history. If you have those turned off, Google does not record that data, and there would be no way for them to derive your location even in theory.