You could try adding in some simple motion, like raising your hand, before the system would start listening for hot words. Maybe connect a cheap infrared sensor to GPIO and block its view so it only detects motion at or above certain height.