Comment by pizza | Hacker News Reader

Comment by pizza | Hacker News Reader

Trying to build a scalable home 4090 cluster but running into a lot of confusion...

Let's say

- I have a motherboard + cpu + other components and they've both got plenty of pcie lanes to spare, total this part draws 250W (incl the 25% extra wattage headroom)

- start off with one RTX 4090, TDP 450W, with headroom ~600W.

- I want to scale up by adding more 4090s over time, as many as my pcie lanes can support.

    1. How do I add more PSUs over time? 

    2. Recommended initial PSU wattage? Recommended wattage for each additional pair of 4090s?

    3. Recommended PSU brands and models for my use case?

    4. Is it better to use PCI gen5 spec-rated PSUs? ATX 3.0? 12vhpwr cables rather than the ordinary 8-pin cables? I've also read somewhere that power cables between different brands of PSUs are *not* interchangeable??

    5. Whenever I add an additional PSU, do I need to do something special to electrically isolate the PCIe slots?

    6. North American outlets are rated for ~15A * 120V. So roughly 1800W. I can just use one outlet per psu whenever it's under 1800W, right? For simplicity let's also ignore whatever load is on that particular electrical circuit.

Each GPU means another 600W. Let's say I want to add another PSU for every 2 4090s. I understand that to sync the bootup of multiple PSUs you need an add2psu adapter.

I understand the motherboard can provide ~75W for a pcie slot. I take it that the rest comes from the psu power cables. I've seen conflicting advice online - apparently miners use pcie x1 electrically isolated risers for additional power supplies, but also I've seen that it's fine as long as every input power cable for 1 gpu just comes from one psu, regardless of whether it's the one that powers the motherboard. Either way x1 risers is an unattractive option bc of bandwidth limitations.

pls help

1. You can pair normal atx PSUs for the motherboard/CPU and server PSUs for the GPUs using breakout boards.

2. You can power limit GPUs down to 250W and barely lose any performance depending on your use case, highly recommend it. So any PSU that can provide those is good.

3. HP 1200w power supplies are both plenty and cheap on ebay, even though they are rated at 1200w, because they are so cheap, you're better of just running them at ~500w and buy multiple of it instead of overheating a single one. A nice benefit of running them at lower wattages is the very loud tiny fan doesn't have to spin as hard and create a ton of noise.

4. Not needed, but having a single cable might be convenient, they are pretty expensive though.

5. You don't need to do anything special here, except if you add too many GPUs, the motherboard might have issues booting because the 75w per gpu draw is too much, but usually those motherboards will have an extra GPU power cable (like the ROMED8-2T) and some risers let you hook up the power cable directly to them so PCIe is only used for data transfer.

6. It's not the outlet, it's the circuit that matters. And keep in mind that whatever power wattage you set on the GPU, you need to account for ac/dc loss, so you need to add an additional ~10-15% to the usage.

If you power limit it to 250W, each additional GPU is essentially an extra ~280W or so. If you plan on having like 8 GPUs or more and you plan to run them 24/7, you're better off just calling a local colocation center and run it there, since they have much cheaper electricity cost, it comes out cheaper for you and you have all the benefits of being in a datacenter.

> 6. North American outlets are rated for ~15A * 120V. So roughly 1800W. I can just use one outlet per psu whenever it's under 1800W, right? For simplicity let's also ignore whatever load is on that particular electrical circuit.

You're going to have a bad time with this assumption; typical non-kitchen household circuits in the U.S. are 15A for the circuit. Each outlet is usually limited to 15A, but the circuit breaker serving the entire circuit is almost certainly 15A as well; one outlet at maximum load will not leave capacity for another outlet on the same circuit to be simultaneously drawing maximum amperage.

Typical residential construction would have a 15A circuit for 1-2 rooms, often with a separate circuit for lighting. Some rooms, e.g. kitchens will have 20A circuits, and some houses may have been built with 20A circuits serving more outlets / rooms.

So those miner motherboards with the crap ton of PCIe x1 slots typically have a molex connector on the motherboard for each of those slots. Molex is famous for starting fires. I’m not sure I would ever go with a setup with molex connectors, but then I’m not sure you have another option. The issue is if they used PCIe power connectors instead, you often wouldn’t have enough of those left over for your GPU, so I get why they went with molex, it’s just a very old, and by modern standards crappy connector.

Combined with the ~1800W per 15A circuit restriction (I wouldn’t load the circuit to 100%, so really ~1600W) I’m not sure you can achieve what you’re going for.

If you’re really wanting to do this, consider adding a say 30A circuit near the circuit breaker of your home, usually the garage or basement and put the equipment there. I would get a dehumidifier in either location.

Have you read Tim's guide?

Reply on news.ycombinator.com