That's just one of several privacy concerns. It's possible to parse voice locally, as in TFA and eg Mycroft('s open source, self-hosted version, anyway), but for "some reason" mainstream assistants don't do it. Sure, you can hold a button, and Google will only hear about your timer request and nothing more, but some people find the idea of Google knowing when you're setting timers to be upsetting. Or at least worthy of avoiding.