It seems like an adversary who could swap your rng could install a radio broadcaster between your keyboard and your pc or reboot to a rootkit.
Oppositely, you might paint your rng with sparkles or other unique paint schemes to mitigate that particular switch risk but then you'd still have the other possibilities.
Anyway, I don't think this kind of physical spy versus spy stuff is the main risk a custom rng is intended for.
That said, if you have an adversary that has and is devoting the time and resources necessary to arrange to have someone on the plane with you, with a doctored USB RNG, in a seat where they can get to your laptop when you go to the bathroom and mess with it, you probably should not be traveling alone, and your traveling laptop should be one that never contains important secrets and is never connected to a network that has important secrets.