It would be trivial for someone to record you saying it without your knowledge, to unlock your phone at a later date.

It does apparently get "trained" on your voice, however do you really want to shout your passphrase at your phone?