https://huggingface.co/unsloth/Mistral-Large-4-1T-A52B-GGUF
I have no idea how you could shrink this down to fit on any type of prosumer hardware though. I doubt it's possible.
I have no idea how you could shrink this down to fit on any type of prosumer hardware though. I doubt it's possible.
This is the stuff I've put together and it is clearly too small for this:
https://jacquesmattheij.com/local-inference-with-large-langu...
The RTX PRO 4000's can still be had at the same price I paid for them, ditto the switch backplane, the enclosure, the PSU and the 2nd hand 3090's as well as the 4090's. What has gone up (considerably) in price is the regular RAM.
"PLX88096 PCIe 4.0 Switch GPU Expansionbackplane 4/5 Slot SFF-8654 SlimSAS"
Is the description on the page. That C-Payne switch is a beauty, they make some very nice hardware.
This is also interesting: