Those fears will be reasonable when most computers will not allow disabling Secure boot. Right now that's not the case for x86.
To give an analogy, there might have been a time in East Berlin where people could have said: "It's fine that they check papers before crossing into West Germany. Your fears will be reasonable when they no longer permit entry."