In the end, turns out that the version of AMD Software and their included drivers from 2023 didn't work and I had to revert to an older version. I ended up going for their 2020 AMD Pro drivers and software because the card is from 2018 and the improvement was immediate - I haven't had a single crash since, whereas previously I could reproduce it 100% of the time, with only the time before the crash varying. Note: this is on Windows 10, which I boot into for gaming, though I actually like their AMD Software which feels a bit better than CoreCtrl on Linux distros.
Some people suggested that 2022 drivers/software would also work, but this is one of those cases where the hardware technically gets updates, but they actually make things worse, perhaps because the older hardware is not tested against as much, especially with newer games and programs. I don't necessarily expect anyone to care that much about hardware from 2018, it was just an interesting case of things going wrong.
Overall, I like AMD because they're affordable (still running a Ryzen 7 1700 CPU on my main workstation and it does everything I need) and decent, and perhaps the driver situation on Linux is a bit better than Nvidia's, but then again I also like Intel's attempts with Arc, except that Nvidia still wins out because of CUDA for a lot of workloads (I never managed to get ROCm working locally well for running LLM's and DKIM kicked my butt, needing to compile stuff after every apt upgrade).
Oh well, here's hoping that we keep getting lots of options for hardware, software and that the market competition stays alive and well, so I can enjoy affordable last gen options with most of the issues patched out a few years down the line as well, when I decide to upgrade on the cheap. But also even in regards to entertainment like gaming things are getting better thanks to Proton. I look forwards to the day when I can 100% switch over to Linux.
My homelab servers already run Debian/Ubuntu with AMDs 200GE CPUs, which I got for dirt cheap and which have relatively low TDP (35W) while still giving me access to x86, which I can passively cool.