If you haven’t unlocked your phone by the time you’re looking at it you’re doing it slower than I am. Typically my phone is unlocked before I’ve even fully pulled it out of my pocket. The muscle memory for thumb (or finger placement if it’s on the back) is a pretty low barrier to usability, and can be learned quickly.
Can you unlock your phone without moving it in front of your face? I can, if it’s sitting across my desk I can unlock it without picking it up or moving my head towards it. I do this regularly.
Perhaps you’ve forgotten how many usability compromises you’ve made in using Face ID, but I guarantee it’s slower.