You should consider trying to retrain your voice model. When I first set up my phone to only recognize my voice, there was people talking in the background and it actually made it behave exactly how your describing.

Shortly after I retrained the voice model in a quiet room by myself and now it works flawlessly.