Not you IP address but your router/access point's MAC address and been doing it for many years.
For example, I receive quite a few devices for reviews, and I mostly give things away after using those. Many years ago, I gave a friend in another city a router. For a few weeks his computer and other devices reported his location as being my city instead of his. It took a couple of weeks for Google services to catch up.
Google logs your current location and the MAC address of any access point visible to your phone. In your case, if you used your phone via the VPN for more than a few hours, Google associates that MAC address with the finer location from your phone's GPS. It may not link that data to your account, but it uses it to help getting a GPS location.
If you live in a location with low traffic/human density then it may take longer for Google to change it back.
This information is also used for a GPS cold start. Actual GPS takes a few moments to get a fix, even more if indoors. A GPS get a fix faster if you are using it close to where it was last used, or how fresh the stored almanac data is, otherwise it may not find expected satellites in certain locations in the sky. While a fix is not available phone GPS uses any visible MAC address to determine its coarse location.
On laptops and desktops with no GPS, browsers will use this access point data to set a location - IP addresses databases are not up to date enough and sometimes they are linked to the ISP office address and not to specific towns and cities and certainly not to a specific address.