I'm working on making my own FPGA based USB 2.0 HID device that will let me minimize any latency in the stack. PCB layout is mostly done, I'm working on a DECA FPGA board to prove out the USB hid software now. I started this pre covid though when Mach XO2s were inexpensive and available, so I have no idea who I will need to fight these days to get parts when I get to that point.
There is lots of wiggle room too in how you implement your debounce algorithm to optimize latency too. I'm excited to control the whole stack to try to make this as fast as possible. The Logitech lightspeed products came out after I started this project though and are far more practical for most people. I have one of those at home and will try to benchmark and compare them when I get there.