You can also explicitly disable the AES-NI in Veracrypt and use pure software implementation if you don't trust the hardware. I usually enable this option.
On the other hand, trying to backdoor XOR would give you an astronomically higher amount of white noise. Finding important data mixed in with all the other things a computer XORs would be much harder.
If the attacker has access to your computer, you've already lost. If the attacker built your computer, it was never even a fight.