The bottom just fell out of the used GPU market. NVidia Tesla 80 24GB on sale for US$79 in quantity. [1]
[1] https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2380057.m...
[1] https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2380057.m...
In terms of ROI, the 3060 with 12GB of RAM seems to be the best one for cheap. The M40 has 24GB of RAM too but the core is super slow.
I'm personally holding out for prices on 3090s to crash. Also, NVIDIA is rumored to be launching their new GPU series in 2 weeks (there is a scheduled event already).
However here in Germany they still cost around 410-450€ used, not $350.
Even the cheaper and recently re-released 12 GB RTX 2060 could be a good pick although the 2060 is generation behind and is less efficient in terms of electricity consumption.
Of course with more GB you get higher resolution images, but plenty of people just generate at 512x512 and then use other AI projects (for example "real-ESRGAN") to later upscale their images, which will still let you achieve great results.
I think a good advice could be to join Stable Diffusion Discord and talk to other users sharing their results and experiences there.
By the way, IIRC (please double-check the following) it could be also worth noting that the guys behind Stable Diffusion (Stability.AI) declared that in the end they will eventually bring down the VRAM usage to approximately 5 GB.
However more GB are always a good thing in general for Machine Learning...