TrustZone is essentially an ISA extension, similar to Intel's TXT and SGX to provide a trusted execution environment. You can trivially avoid it by never running any of the related instructions.
It's very very different than the dozen or so instructions to setup TXT or SGX that sits off to the side of the main OS rather than running like a super hypervisor. If you're going to compare it to something, it's way more like SMM on x86.
Source: I've ported a kernel to EL3 (secure mode).