The AMD MI250X GPUs are also interesting - 128GB of HBM2E at 3TB/s, sometimes you see them second-hand for under $1k, the catch obviously is that it needs an OAM socket. Never seen an easy way to hook them up to a regular mainboard.
Also, the cheap HPE pulls on eBay need some proprietary HPE magic to work, and I have yet to see anyone figure that out.
It fits an MI250X, and the system sees it, but the drivers don't work. They tested an HPE MI250X. There's a rumor on the thread that there are two kinds of MI250X: Ones from HPEs and everyone else's. The HPEs require a special firmware, the normal ones do not. However, the majority of the MI250Xs on the secondhand market are HPE so caveat emptor.