And on the other side, projects like stalwart email ( http://stalw.art/ ) and mox ( http://xmox.nl/ ) makes it so much easier to build an e-mailsystem on your server.
81 karma · joined October 25, 2016
I never reballed any IC, so I would not dare to try it on my card.
There is a nice coverage on this topic at https://www.youtube.com/watch?v=Tld91M_bcEI (Why the Original Apple Silicon Failed)
7x RTX 3060 - 12 GB which results in 84GB Vram AMD Ryzen 5 - 5500GT with 32GB Ram
All in a 19-inch rack with a nice cooling solution and a beefy power supply.
My costs? 1300 Euro, but yeah, I sourced my parts on ebay / second hand.
(Added some 3d printed parts into the mix: https://www.printables.com/model/1142963-inter-tech-and-gene... https://www.printables.com/model/1142973-120mm-5mm-rised-noc... https://www.printables.com/model/1142962-cable-management-fu... if you think about building something similar)
My power consumption is below 500 Watt at the wall, when using LLLMs,since I did some optimizations:
* Worked on power optimizations and after many weeks of benchmarking, the sweet spot on the RTX3060 12GB cards is a 105 Watt limit
* Created Patches for Ollama ( https://github.com/ollama/ollama/pull/10678) to group models to exactly memory allocation instead of spreading over all available GPUs (This also reduces the VRAM overhead)
* ensured that ASPM is used on all relevant PCI components (Powertop is your friend)
It's not all shiny:
* I still use PCIe3 X1 for most of the cards, which limits their capability, but all I found so far (PCIe Gen4 x4 extender and bifurcation/special PCIE routers) are just too expensive to be used on such low powered cards
* Due to the slow PCIe bandwidth, the performance drops significantly
* Max VRAM per GPU is king. If you split up a model over several cards, the RAM allocation overhead is huge! (See Examples in my ollama patch about). I would rather use 3x 48GB instead of 7x 12G.
* Some RTX 3060 12GB Cards do idle at 11-15 Watt, which is unacceptable. Good BIOSes like the one from Gigabyte (Windforce xxx) do idle at 3 Watt, which is a huge difference when you use 7 or more cards. These BIOSes can be patched, but this can be risky
All in all, this server idles at 90-100Watt currently, which is perfect as a central service for my tinkerings and my family usage.
and mostly unhealthy in comparison to self hunted / self-planted food.
I took a fair amount of time to get everything to a reduced power level and measured several llm models (and hashcat for the extreme) to find the best speed per watt, which is usally around 1700-1900 mhz or limiting 3060 to 100 to 115 watt.
If I planned it in the first run, I may got away with a used mac studio, thats right. However, I incrementally added more cards as I moved further into exploration.
I didn't wanted to confront someone, but it looks like you either show of 4x 4090 or you keep silent
Oh nice! So I can try it in my local "low power/low cost" server at home.
My homesystem does run in a ryzen 5500 + 64gb RAM + 7x RTX 3060 12gb
So 64gb RAM plus 84gb VRAM
I dont want to brag around, but point to solutions for us tinkerers with a small budget and high energy costs.
such system can be build for around 1600 euro. The power consumption is around 520 watt.
I started with a AM4 Board (b450 Chipset) and one used RTX 3060 12gb which cost around 200 Euro used if you are patient.
There every additional GPU is connected with the pcie riser/extender to give the cards enough space.
After a while I had replaces the pcie cards with a single pcie x4 to 6x PCIe x1 extender.
It runs pretty nice. Awesome to learn and gain experience
It should be enough to fix that issue
If you haven't heard of Asianometry, give it a try!
SCNR!
It's good
> I have male friends
It's okay
> My boss is female
It's good
> My boos is male
It's okay
I am pretty sure there are far more magazines/podcasts/blogs which work with just supporting readers, but you have to find them.