Your cost/benefit considerations prioritize relatively miniscule security improvements without considering usability costs or diminishing returns. While we're at it, why don't we just use one-time pads? After all, those are impervious to any form of cryptanalysis.
The risk profile for most users does not require a hardware-based auth factor if it results in real world usability sacrifices that end in either 1) accidental misuse or 2) gradual disuse.
You're optimizing for someone compromising the device, great. But the point is that if that risk if on the table, all of this work is essentially meaningless anyway.