Moving my PC into my rack in a 2U case
jeffgeerling.com
jeffgeerling.com
That old server has gone through many, many incarnations of hardware.
https://i.imgur.com/S5F31qW.jpg https://i.imgur.com/SpMMPXQ.jpg
The magic had already left the old Sun hardware, when we tried it.
I really regret that decision.
Bookshelves, desks, old and new machine tooling, gigantic wrenches
Though if I ever buy I house I might end up going Linus level crazy and put a rack in the basement or a utlity room with fiber optic cabling back to my desk for peripherals to attach.
For UPSes, a reputable brand (like APC) will still function fine but might need new batteries, which are a lot cheaper than a whole UPS.
I haven't shopped rackmount UPS before but for lower end consumer APC UPS the official batteries from Amazon or the APC website are often the same price or more expensive than a whole new APC UPS from a slickdeals-spotted sale.
Double-edged sword. Higher prices brand new, but vastly cheaper used if you chase companies' deprecation cycles.
It's often the most cost effective way to get redundant and A+A power supplies in servers that you need to protect beyond simple UPS methods.
Source: Loving the 2U and 4U blades with 2017-2020 era EPYC and Xeon processors with dirt-cheap Tesla P40 24GB VRAM GPUs in them for inference and all sorts of compute loads.
https://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ss...
Then by that definition, virtually every blade server I've ever seen was 2U :)
> Small servers like that are also very unlikely to be loaded up with high-watt GPUs for inference or training workloads.
True, though I've seen full-size GPUs get squeezed into form factors much smaller than the average 2U-high blade.
Huh?
Rack Us absolutely refer to a form factor not just a z-height. It would make no sense to refer to something in # of Us unless you either want to then mount it in a rack, or be deliberately annoying.
Most people (including, typically, the companies actually manufacturing and selling blades and enclosures) would also in this context reasonably infer "2U or 4U blade" to be referring to a (partially-or-fully-)filled enclosure, rather than pedantically assume that someone (who is implied to own and use such a server) doesn't know the difference between a blade and a pizza box - but given that we've already deviated from using terms the way anyone else uses them, what's one more deviation, right? :)
The entire point of blades is that they’re modular and a chassis can be filled with many different kinds of blades. Calling an enclosure a blade would be like calling a tire a car.
Source: I’ve specd out and ordered literally thousands of blade servers and hundreds of enclosures from every major vendor.
Edit: an even better simile would have been people who call a whole desktop computer a cpu.
there's a logical reason for this though. it's a pretty safe assumption that you're in a larger facility if you're rack mounting. a lot of rack mount chasis come with features like dual PSUs and other redundant features. redundancy does cost.
>put a rack in the basement or a utlity room
They make enclosures[0] that are smaller than a full rack, but allow for rack mounting equipment. you can then just shove this enclosure on a shelf or wherever. much more convenient than a full on server rack.
[0] https://www.amazon.com/Wall-Mount-Server-Rack-Cabinet/dp/B07...
And the reason I would put the equipment in another room would be to keep the heat away from the living space. A basement would be especially good for that since it's natually a cooler area.
Still good to know about the smaller racks though. My bank account is much healthier than the last time I looked into rack mounting so I might actually get one that size for my apartment now. It would really neaten things up.
Also, even if you don't have rack mount equipment, they make the shelves you can attach to the rack mounts and set other gear on them. So yeah, you can really clean some stuff up with these.
These cheap wall mount units including the one you linked are great for rack mount network or AV equipment, not so much to rack your PC build like Jeff did here.
The width for server racks is standardized - the depth however varies enormously from rack to rack, with wall units usually being pretty shallow. I'm not sure most people would even want something as deep as a say a typical 2nd hand 1U/2U blade server hanging on the wall - they often need 29 inches or more.
EDIT: The box Jeff has used is one of the few I've seen that might actually work well in a wall rack, if you can live with an mATX form factor motherboard in your build, at ~9 inches, which leaves enough room for power and air at rear.
If you want a Linus Tech Tips style gaming rig or mid/fullsize tower build in a rack, I still think you might need to go free-standing most of the time.
I found the batteries in the rack version, where batteries lay on their sides, reach end of life much sooner than the tower version, where batteries are upright.
It's likely poorer heat dissipation/ventilation in the rack version, but may have to do with battery orientation. The rack version also seems to suffer corrosion damage from battery off-gasing if batteries reach critical failure and overheat. I haven't seen the same damage on the tower units. Again some combination of orientation and ventilation.
My consumer machines are generally flaky as hell in comparison.. maybe partly due to windows and gfx card drivers? Ymmv.
Typically, what determines the life of these batteries is # of cycles and # of deep discharges. The more extreme the environment the shorter the life time. I have network equipment sit outdoors in a cabinet. The batteries typically last 24-36 months instead of the 48-60 months I see from indoor UPSes.
I recently had a Supermicro board die because it became infested with moths out in the garage, it was disgusting and the board was borked even after extensive cleaning.
How do you keep wildlife at bay? (insects and rodents are attracted to heat and shelter).
Maybe some kind of solid metal cabinet enclosure like the public utility boxes along some roadside thoroughfares?
Outdoors you need special enclosures typically made for industrial computers; and many companies build special fanless enclosures that dump heat directly through the metal enclosure (it is basically the heat sink).
Others have fan designs that have their intake and exhaust through filters on the bottom.
Otherwise, search online for 'PC dust mesh filter' and it usually is sold in rolls.
You don't need strictly rackmount hardware either. I just have a pair of non-rack SUA1500's sitting next to one another on the solid bottom built into the rack (not even a movable shelf). The battery type is much more common than the rackmount versions.
Case wise I'd just recommend 4U all the way for full size cards, coolers, and fans. Smaller fans make more noise.
Also just spend the money on cases with hot swap disk bays for any machine you plan to have more than a few drives. You'll end up wanting them sooner or later, and it's nicer to run the SATA cabling once with how cramped rack cases can get in places.
no way, are you telling me that a $180 rackable shelf is a scam?
Second hand (but perfectly usable) parts. Expect more second hand enterprise items to get to the market if the economy doesn't start to boom soon. Second hand enterprise gear is usually perfectly fine in terms of longevity. Mind the noise.
Get a 3d printer, print some rack adapters yourself. There's a surprising array of adapters for all sorts of devices and they work just fine for stuff that's not too heavy(like most home networking gear).
If the rack itself is too expensive, you can do an Ikea Lack rack. Full size racks are easy to find (relatively, location dependent). Haven't had much luck with network racks (which was the largest my wife was comfortable with). So I bought the rack new. Still managed to fit the whole networking gear stack(none of which is racked atm, they are sitting on a rack shelf) and a ITX server (Node202, which I used as my workstation in the past, fit perfectly).
You don't really want fiber optic unless you are connecting two different buildings that do not share ground. Inside the same house, ethernet works just fine. Fiber is finnicky, most home contractors don't know how to deal with it, and transceivers are expensive. More expensive than 10G usually.
In the home studio world this kind of setup is much more common. I think a lot of people are attracted to the aesthetic of rack-mounted studio gear. I know I definitely am. I personally don't find this setup as ergonomic as desktop modules though. I think manufacturers are coming to the same conclusion.
I could have used wifi, the bandwidth would have been fine, but where's the fun in that ;)
Going another step, I'd love to have a Thunderbolt to fiber adapter without having everything baked into a single cable. These products exist, but cost $5000 for each end
https://www.atto.com/products/thunderbolt-adapters/thunderli...
On a similar note I recently worked with an adapter that carries USB 2.0 over a cat5 cable. It doesn't even do that good of a job -- jitter is pretty high, which makes it useless for our use case of controlling a laser cutter live from a host computer -- but guess how much it costs? $736! https://www.blackbox.com/en-ca/store/product/detail/usb-20-e...
But that became kind of limiting for my preferences. For a while it wasnt really practical, unless you could accept this pretty low bandwidth stream.
But now fiber-optic display cables are readily available. My first purchase was a 100ft/30m 32Gbps DisplayPort cable, for $55! To be honest, I couldnt believe it worked; I assumed it was a scam.
It's indeed excessively long... too long! I can make it out my room & up to the roof & compute from there with less than half the cable. It's way harder to unroll & reroll than any cable I've ever had before, but also way longer. I have since gone back and bought shorter cables.
Thankfully active extension cables are pretty good & tend to just work. Newer models have an inline 1-port usb-hub every 25ft/7.5 (used to be every 5m), and then a wall-wart at the end to power the hubs (to honest that feels like it should be unnecessary; a dc-dc booster mid-span & bus power should be fine, albeit leaving nearly no power available at the end).
Also beware, there's a 7 tier limit from root-hub to device, so 5 hubs maximum * 7.5m, if you only plug in a single thing (a wireless keyboard/mouse dongle).
Latency would be super interesting to test for both. I tend to think 4-5 usb hubs are still under 1ms latency. I'd expect the DisplayPort digital->optical->digital path takes less than 1ms too. But I'm guessing, for sure.
SunRays (https://en.wikipedia.org/wiki/Sun_Ray for those too young) were really nice. When they came out I realized why have a noisy workstation under my desk when I can have a massively larger (far noisier) server in the rack and a totally silent SunRay on my desk! That was my setup for a long time.
No longer using the SunRay (sigh) but conceptually not too far from it. I have a macmini on my desk but I do very little compute on it, it's there basically just to drive the monitor. All the machines are on a rack in the garage.
Eventually found someone selling an old 4U (which is about as tall as a standard desktop is wide) ATX server case, locking front panel and all, on craigslist for cheap.
It was super heavy (as was all of the other audio equipment), but having all of my stuff in one case (and only needing to set up a display, keyboard, and mouse) was pretty great. Never had much issue with heat/noise, but wasn't overly concerned at the time either.
[1]: https://www.pssl.com/products/gator-console-rack-10-top-6-bo...
One interesting thing about going the rack route is that most of the hardware optimizes for considerations a data center/large corporation might have.
That can be nice in a lot of ways (hardware is fairly robust, layouts are usually friendly for quick access/maintenance, overall density of machines is pretty good).
But it means almost no one is paying attention to how much noise these things make, because for most of the target audience that's WAY down the list of things they care about.
I have a couple of - not elderly but not new - 1u and 2u machines, and they are LOUD AS FUCK. I used to run my server farm in my office when it was just spare personal computers (under the table I use for hobby projects), but I quickly ended up moving it down the basement when running the rack machines.
They're just too loud for comfort on a semi-regular basis.
There are fans mods and firmware updates, but in the end it’s probably the best use of one’s time to buy G9+ as they are far quieter. /r/homelab has some great data about these issues.
Also watch out for network gear. A 48 port POE brocade switch was as loud if not louder than the servers.
Pro-tip: if you can spare more rack units, get a bigger server case. 1U is usually far louder than 2+
Otherwise, manual soundproofing is an option, but then you have heat to deal with.
Most servers aren't very loud unless you're stressing the hardware, and fan settings can usually be tweaked, or fans replaced with silent alternatives.
Still, you wouldn't want this kind of equipment humming next to you in the same room, so it's best to dedicate a small server closet, room, or garage, if you have the space.
That said, a custom server build in a rack mounted enclosure is not the same as using enterprise servers. My gaming PC is in a 4U enclosure and is as quiet as any tower PC.
The hot row/cold row stuff makes sense when you've got a data center, but if computers are eating the world then more and more people will need 80u of rackspace total, maybe at 3 branch offices.
and the smaller the fan, the more RPMs (noise) needed to move the same quantity of air.
The problem is that rack mount plus low noise equals very high cost, if it is even achievable. Eventually I could not stand the noise anymore, and got rid of the rack. So did every single one of my colleagues who tried rackmount.
As others have mentioned rackmount equipment is also expesive, often ridiculously so. I am very happy to leave racks in the data center where they belong.
I have a friend who keeps his server rack in his unheated detached garage. Works great unless it gets really cold and all the servers shut down due to low input air temperature. Which I never really knew was a thing.
And now that you can get heat pump water heaters, that strategy would work a bit better still.
Here is my AIO cooled threadipper in a 4U enclosure
I have a 3U setup where the loudest thing in it is the GPU fan on the 1050Ti.
They relocate the noise to where you aren't, so they do address it.
As I mentioned in another comment, my desktop used to be a big Sun server (forgetting model right now) loud as a jet engine. But it was in the rack far from anyone and my desktop was a totally silent SunRay.
My second stroke of luck was that the hall closet in my apartment had a corner in it that would exactly fit this rack, telephone wiring junction box was immediately above the rack, and all of the house wiring was straight Cat5 runs from there, so a few new outlet covers and some cable rejigging and we had internet everywhere (but only 2 pair where the phone plugged in), terminated at the rack, which was partly muffled by winter coats. You can't push a lot of BTUs this way, but you can do a few.
My solution before this was to hide a tower PC behind the couch. For the right kind of couch and the right vent holes, you can eat an awful lot of ambient noise by using major furniture. There's lots of space out there for things like this.
I also tried at one point to build an air filter that would fit under a bed (less noise, no floor space, fewer dust bunnies.
I run proxmox, opnsense (virtualized) and other workloads on it. It's really good and runs almost completely silent because of the Noctua fans.
The build quality of the case is lower end though with sharp edges!
The downside is that they are 4/5u. But they are a fucktonne quieter than server hardware.
The difficult part is getting the HDMI/DP extenders to work reliably or low latency enough without spending $2k.
Having said all that, Jeff's approach is most likley cheaper, if more fiddly. its certainly neater and more compact.
on the pricier side if you have something with TB support, you can do display and USB over that to a fiber optic TB cable (about $400); this is my current setup for my racked PC in another room to my office, with a TB hub (caldigit TS4) that splits out display and USB.
when I last did this it was for a VFX house. They used a fibre "sender" and for really long distance they used a teradici system.
*Larger fans running slower for less noise
Nowadays my desktops are either SFF or mid-tower CAD workstations. We've got some control computers racked with test equipment/tools but they're not really general purpose workstations.
The only I've used and like goes out of stock regularly though.
Though I think you could get a double-wide GPU into a 2U with a PCIe riser cable/card.
You can also spring for their matching drive bays http://www.istarusa.com/en/istarusa/product_speclist.php?ser...
https://www.rosewill.com/rosewill-rsv-l4500u-black/p/9SIA072...
- Full-height PCI cards fit (though some GPUs have taller than full-height coolers; if the HS/F protrudes above the mounting bracket, you'll have to measure).
- You can use larger fans, which can help a lot with airflow and/or noise. Just throw away the fans that come with the case as anything designed for a rack assumes you'll be wearing hearing protection while working around it.
- More motherboard options. If you use a power-supply designed for rack-mount, standard ATX motherboards fit fine. If you want to use a desktop power supply there are plenty of micro-ATX options.
Though, I did it in a 2U optical cross box. I paid $30 for it, and replaced some panels with fans and grilles. Not 200 euros, holy moly!
I paid almost £260 iirc for a 4U case with 16x3.5" hot swap bays, a few years ago, and that seemed like a great deal (without waiting for something used on eBay anyway). Now, I still haven't really made use of it... but that's on me.
My Ryzen 5800X3D I think has issues with c states which caused reboots from low to high power states.
Thought it was the 3800RTX and 600w PSU. Upgrade SFX-l which has you loose the rear 120mm fan.
Whole thing really started to cook itself till I disabled the c state and switched back to the 600w
I do miss ATX because ITX boards generally do not have any expansion slots. Sure they come with Wifi, but if I decide to, say add a 2.5G(or 10G) network card, video capture or what have you, I'm out of luck.
I do not miss the case sizes. ATX are spacious, but TOO spacious, usually. Because the mainboard is large, most cases don't seem to optimize for space at all. So you fill the thing to the brim with all expansions you can think of and there's still a lot of wasted volume. Were I to go back to ATX, it would be with a non-orthodox case(open case, inside a desk, etc).
(I don’t have room for a rack, but I do have a moderately tidy server closet with an UPS, and remote to these machines with GPU-accelerated RDP.)
*Fitting the GPU*
A normal GPU, mounted in the normal way (perpendicular to the motherboard), will not fit in a 2U case.
It looks like Jeff's solution was to buy a half-height GPU. Neat! I didn't actually realize those exist. But it looks like NVidia's regular consumer-oriented GPUs don't come in this form factor; Jeff instead went with an RTX A2000, which is part of NVidia's "professional" GPU line. The problem with this is that it's way more expensive to achieve the same gaming performance.
In my previous house I solved the problem by using 3U chassis. This provides just enough space to mount the GPU perpendicular. However, it still had problems. Modern GPUs require supplementary power direct from the PSU. The socket for this usually points outward, away from the motherboard. In a 3U chassis, this socket is right up against the top plate so there is no room to plug in the power. Sometimes you can find a GPU that has the power socket pointing in a different direction, but they have become rare. I actually had to physically mod ten GTX 1060's by hand, cutting away the plastic socket housing and directly attaching each power wire to the corresponding pin. Big PITA but it worked.
In the new house I'm going with 2U, but using a riser card. So, the GPU is mounted parallel to the motherboard. This seems to work OK. The big problem is ensuring the chassis and the motherboard both have their expansion slots aligned right for this. Most 2U chassis are designed for perpendicular, half-height expansion cards. A smaller fraction support full-height parallel cards on a riser. The motherboard also has to have a PCIe slot in the correct place for the riser. This is usually the "top" slot, closest to the CPU, and I've noticed a lot of motherboards these days are opting to omit that slot to make more space around the CPU.
The chassis I'll probably go with is the iStarUSA D-200: https://www.istarusa.com/en/istarusa/products.php?model=D-20...
(I'm not too excited about it though, and would be happy for other recommendations.)
At least one motherboard that worked when I tried it was the ASUS PRIME B660-PLUS, but I haven't made a final motherborad selection yet.
A further problem which might be specific to this chasis is that the total horizontal space between the riser card and the power supply is tight. Some GPUs are excessively "tall" and won't fit. (The same problem probably applies to 3U chassis with perpendicular mounting.) I was just barely able to fit the EVGA GeForce RTX 3070 Ti 08G-P5-3785-K into the space.
*RAM orientation*
Servers orient their RAM modules parallel to the direction of air current, that is, perpendicular to the back of the chasis. For reasons I don't understand, consumer / gaming motherboards almost never do this, they orient the RAM parallel to the back. Maybe this is because desktop chassis are not usually optimized for efficient airflow anyway so it doesn't matter? And there's more space on the board if the RAM can be mounted this way? I dunno.
Anyway, in my experiments so far this has just not caused a problem. Yeah, the RAM has non-ideal orientation, but there haven't been any issues with overheating. ::shrug::
*PSU*
Servers use a different form factor of PSU. This is infuriating because all the server PSUs available retail seem to be terrible compared to desktop PSUs! They tend not to support higher-end wattage (needed for power-hungry GPUs) and they are never "modular" (you get a bundle of cords sticking out of the PSU and there's no way to remove the ones you don't need except with scissors).
There are some 2U chassis that actually allow you to mount a desktop PSU. However, in practice they are not actually compatible with any modern desktop PSU, because all such PSUs these days are designed with their intake fan on the top or bottom, not on the front/back. In a 2U chassis, that intake will be totally blocked by the top/bottom of the chassis.
I am going to have to live with crappy server PSUs I guess.
*Retail server market is awkward*
More generally, I get the general feeling that the server-oriented components which are available via normal retail channels like NewEgg are... not what big server farms actually use. Everything seems kinda crappy? Maybe I just feel this way because they don't candy-coat everything the way the consumer market does. But I have the funny feeling that people building out datacenters generally order mass quantities of chassis, PSUs, and even motherboards tailored specifically to their needs, and as a result the retail market for server components may be surprisingly small and poorly served. But that's just my suspicion based on my experiences here.
Not all server chassis obviously, but many have fans mounted at the front and pull air through the case, over the motherboard, through some plastic ducting, and out the back. Some server designs have only a heat sink on the CPU, and rely on this ducted air flow from the case fans to keep everything cool. RAM header orientation will often be due to CPU, VRM and memory controller orientation, but mostly it is about real-estate on the motherboard and physics.
Retail market for server chassis is pretty abysmal. You might consider a Cooler Master, or a SuperMicro or an ASUS case. Depends on your budget and your willingness to use a Dremel.
If you are considering a 2U rack mount case with a full-sized GPU, you might want to look at cases that specifically mount the GPU sideways, there are quite a few out there that will happily take two GPUs, though the cases are often full-depth. Just be aware that server GPUs and consumer grade GPUs have different cooling options and consumer grade cards may not fair well if the case is too close to the exhaust fan.
Instead of a riser, and fighting for a PCIe slot, you might also consider a PCIe extension cable, and then fabricate a custom bracket to hold the GPU in the desired orientation.
The closest that exists out there is probably SuperMicro. They have a decent variety of chassis, backplanes, motherboards and so on that work well together in a system, and the stuff is pretty decent quality. But because they are still designed for 'serious' workloads, they will not hit the cost of an equivalent desktop PC.
This is surprising to me. I really enjoy 4k for the better text readability.
They seem very gaming oriented, because well, they are - but being able to read things as they fly by is a huge help. I'm always the person in calls at work going: "you missed it!"
I have done that though, where you just let loose the firehose, and scan for patterns during some nasty event.
To this day I still try to keep a log feed visible along with the grafanas or whatever - it is shockingly useful even if I don't ever read those logs on the display.
So for now I’ll keep my ancient dell 1080p displays.
there is a new “USB-C Hub” monitor which is 16:10, but also 1920px across from dell: U2421E
https://www.dell.com/en-us/shop/dell-ultrasharp-usb-c-hub-mo...
Though ultrawide does lift my spirits a bit. Doesn't improve the feel of the desktop for productivity, but gaming is excellent (first person games become more immersive and in WoW my screen is so much less cluttered with less important elements pushed way out to the side).
If you have 1080p though you’re already on 16:9, did you mean 1920x1200?
Edit: The second I posted my reply I saw your other one, U2412Ms indeed. A decade on and they're still as good as new. :)
"moving" to me implies that the 3080 was somehow made to work inside that case. regardless, the conversation of this thread has some interesting takes about this concept.
https://www.ebay.com/itm/124039115388
With adapters for the 5.25 bays, this meets the criteria:
https://www.amazon.com/RackChoice-Compact-Rackmount-Chassis-...