As for the specs, I went new Threadripper way simply because that's AFAIK where You'll get the best balance of PCIe 5.0 lanes (for the memory bandwidth between the GPUs) and power.
I see how You could spec this out with some used Xeon, but for one I really wouldn't like the headache and another, You're probably getting PCIe 2.0/3.0 tops (if You're trying to cut costs that much).
That Supermicro You mention seems to be an interesting compromise. The MB is selling second hand for about ~700EUR in the EU. The Threadripper that goes into it... don't know, can't find them used, but guessing something under 1k EUR as well.
Interesting price, but then on top of the risk of using old parts, instead of partitioning PCIe 5.0 (and maybe having the option to go 4.0 and double the count, since Your GPUs are PCIe 4.0 anyway), You're now splitting straight 4.0 lanes to Your GPUs. Half the bandwidth, but it may end up not mattering all that much. Given that most LLMs are memory size / memory bandwidth limited, I probably wouldn't like to risk this. But that's just me...