you could set your password something like "hey siri", which essentially is a keyword to wake siri up.
not often but sometimes, siri wakes up on it's own. i guess people were concerned at early times, but nowadays it's just _another bug_ in the software.
I do not see passphrase (i think the passphrase is a better word for this feature) as a big issue at the moment.