Who owns the key(s)? If not you, how do you trust them? If you, how do you write them? How do you prevent attacker from writing them? Or is it one-time programmable and your hardware becomes brick / unresellable / un-updateable if you lose your keys or don't want to pass them on to next owner? How do you make sure your keys are more secure than the machine that somehow got its firmware infected? What if the code that checks your signature turns out to have a bug? Are you stuck with it, or can you update the code? What if an attacker can infect that code..?
Making this all convenient and safe is very hard and in practice either takes freedoms from you (you must trust your vendor and their signed binaries), or gives a potential attacker the same freedoms that you get. Giving you full access without enabling an attacker to have the same access is quite impractical (read: unlikely to ever happen for consumer hardware that you can order over the internet), and you still have to trust the hardware is what it claims to be when you get your hands on it.
In practice, I can't audit the hardware but I'd be pretty good if I could flash it myself once it's in my hands.