I always assumed that it was a separate i2c bus per HDMI link and that it was the AVR’s job to handle a request from something and send the right requests to everything else.
Version 1.0 and later of the HDMI spec even mandate that you have to connect those pins across all HDMI ports on your device even if you don't do anything with them.
I wonder if a malfunction in this process is responsible for my AVR sometimes auto-switching to the wrong source.