The hardware can do it, it's just that the system settings won't show you the 4K resolution option for some reason. But you can do some hacks to make it appear and then it works just fine.
You need to install a nondescript app called 'Samsung Good Lock' from the Samsung store (not available in Play store), and use that to side-load an app called 'Multistar', which is an app to tweak display settings. From that side-loaded app you need to tap the 'I Samsung DeX' which does various setting changes to "Make Dex even more friendly", it doesn't specify what it does exactly, but it'll make the 4K resolution option appear in the system settings.
This all feels real sketchy and I don't understand why Samsung doesn't just enable 4K resolution officially, because the hardware is clearly capable of it.
With every OneUI update there are rumors that it'll natively support 4K, but so far that hasn't happened AFAIK. Admittedly I haven't used Dex in a while for myself, but judging from recent Reddit posts this hack is still needed.