Sounds like you should get a monitor with built in support for KVM. I can do that on my Benq PD2700U.
This is the first year that kind of monitor exists. :)
By the way, I've always found having a dumb usb switch (for each device) and a dumb video switch to be the best way to eliminate incompatibilities. But everyone wants to get fancy and "smarten up" their kvm to use hotkeys and it screws with latency or keyboard state or just adds weirdness.
So I think the smartest place to put kvm functionality is probably at the keyboard, in the mouse and the monitor.
Or we should have an open kvm solution