In in the end I just went the whole hog and set up a PiKVM, so now if I mess up the machine's networking (or even completely break the OS) I can still recover it remotely even though it doesn't have a proper BMC or anything like that.
In general this approach seems ugly in principle but I really like it in practice. It lets you retrofit solid remote capabilities onto consumer hardware. That way you have such a broader market to buy from.
It's pretty funny that you have both the input and output physically installed in and powered by the chassis, but then you run cables to connect them!
It would seem reasonable to at least have the option to have it directly enumerate as a USB hub/display device on the PCIe bus it gets power from! But maybe that would add a lot of bulk/cost?
I take it you're in the US? Because it seems I (in Europe) could order it from Aliexpress, see the link on https://github.com/sipeed/NanoKVM.
EDIT: It seems the NanoKVM is struggling with one of the most fundamental KVM tasks, though: https://github.com/sipeed/NanoKVM/issues/594
I really liked the PCIe form factor, but it looks like I'll have to go for https://tinypilotkvm.com/ instead.
Of course, the RockPi doesn't give you any KVM like functionality, though.