Have you tried installing rocm? Last I checked two months ago you had to run a Ubuntu 16.04 container on Ubuntu 18.04 because the amd drivers were only compiled for bionic and the rocm software was only compiled for xenial.
https://github.com/nixos-rocm/nixos-rocm
The last time we purchased GPUs, we bought two Radeon VIIs besides RTX 5000s. For training transformers in Tensorflow and PyTorch, they were a steal, because the training is only marginally slower than the RTXes at a fraction of the price.
For LSTMs, we have found performance regressions compared to NVIDIA, but it's largely because it was hard to hit the optimized MIOpen code paths.