If you destroy the keyfile, then even knowledge of the passphrase won't allow you to decrypt the data. This can be used to defeat a rubber-hose attack [0].
Standard security vs. convenience trade-off applies, of course - if you keep the keyfile on a USB key, then if the USB key fails you lose access to your data. If you keep a backup of the keyfile somewhere else, then access to that backup plus a rubber-hose attack to get your passphrase allows an attacker to decrypt your data.