This isn't happening for me, if by "over time" you mean "several weeks" of heavy GPU use without a reboot. Consider the possibility that the root cause is not macOS.
If I avoid doing those things, it can last much longer.
I have the 128GB/8TB M4 Max and I'm not even coming close to using up all or even most of the unified memory. I suspect it may be related to some form of fragmentation, though I have no way to know how.