The only way to opt out is to turn your phone off (equivalently, leave it at home).
For some people, the fact that google also knows it is no big deal. For some it is. But if you are worried about your whereabouts being leaked, how can you justify using a phone in the first place? Verizon and AT&T are more likely to get hacked than google (or just sell that info outright, which they have done wholesale before)