I got a Stable Diffusion thing up and running locally on a relatively old AMD RX 580 with 8GB of memory, and default settings of 512x512 with 10 (I think) passes, and it takes about 10 minutes to generate an image. On a similarly aged Nvidia 1060 with 6GB of memory that a colleague of mine uses for their local Stable Diffusion instance, it takes about 20 seconds to generate an image.
I don't know how apples and oranges this comparison is, my OS is Linux and his is Windows, for example, but 10 minutes versus 20 seconds is, for me, a massively disappointing difference.
I'm sure the gap has decreased in the intervening 6-7 years, but I don't know how much - such that I'd be opting for a second hand Nvidia over a new AMD (and I philosophically prefer AMD).
This may be a worthwhile read if you're looking to squeeze the most performance: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...