It's a really interesting point. I've had to go through quite a bit of trouble to get non-keyboard HID devices (like Yubikeys) "un-Dvoraked" so that they work as expected.

Interesting thought: Keyboard layout should be a setting scoped to the specific hardware gadget, and everyone does that wrong because it's easier to do it wrong.