Yeah that is really annoying.

You can install https://github.com/MonitorControl/MonitorControl with uses Display Data Channel (DDC) to let you adjust volume and brightness on external monitors. It works for my monitor at least.