That's because you've got location services turned on. There is an opt-out, but it does break Google Assistant.
it does break Google Assistant
From personal experience, it breaks a ton of stuff, including my patience with the endless "turn on location services" prompts.
I turned all assistant features off (search obviously still works) and only location on for apps. Location history and sharing are also off but I'm not sure if they still save some location information. I'll wait for the 25th May and then download data (I'm in the EU) to see what they have..
You now can download what's currently available through Google's Takeout. Maybe once GDPR is in effect more stuff can be available, but I'm not sure what those will be.
That's why I wait. Don't want to do the work twice in one month. It could be that they already provide all data but I'm only sure from the 25th.
I have location history off, but obviously need location turned on for my apps to work. I only want to turn off Google (I even froze the Google Search app).