Could just as easily be your Thunderbolt dock (if you are using one) or adapter.
Most docks and a lot of the adapters out there completely suck.
CalDigit is perfect, though. No issues with my 2 external monitors on my 16" MBP since I got a CalDigit dock. Now I plug in one cable, and everything is kosher: 96W power, GigE, external speakers, 2 external hard drive arrays, and 2 monitors.
Was perfect under Catalina, and remains perfect under Big Sur.