Hardware clocks are famously unreliable and inaccurate. NTP has failure modes that can result in servers being wildly out of sync with reality. Letting either one of those hose your ability to log in and diagnose/recover the system is a mistake.
Would I employ this for a critical commercial server? No. Would I do so for my private server? Absolutely, without hesitation. If it really loses track of time so badly (hasn't happened yet to my knowledge), I can just log in through virtual console.
[1] I'm excluding drift and inaccuracy, since from my anecdotal experience they are usually not that bad.