Sure, if you're hardware RNG is inspecting your RAM and trying to compromise your entropy pool that could be done, but:
1 - The entropy mixing is more complex than simply XOR, making such a thing considerably harder
2 - If you expect this level of backdooring from your CPU, you have bigger problems :)