Hardware SAS Raid on the Raspberry Pi CM4 | Hacker News Reader

Hardware SAS Raid on the Raspberry Pi CM4

jeffgeerling.com

63 pointsgeerlingguy22 comments

> A Broadcom engineer named Josh watched my earlier videos and realized the ancient LSI card I was testing would not likely work with the ARM processor in the Pi, so he was able to send two pieces of kit my way

I'm curious about this; naively, I expect hardware/drivers to be basically independent of architecture/platform. If I've got the connectors/adaptors, why can't I plug in any arbitrary card and have it work?

One problem I've found specific to the Pi (and likely most of the other lower powered ARM chips out there, from what I've seen from Rock Pi, Pine boards, etc.) is that the PCIe layer is not 100% supported on the hardware level as it should be if you want to support features like IO bar space (nonexistent on ARM).

On top of that, until the past couple years, there were only a few exotic ARM builds (probably mostly in custom or enterprise land) that even had standard PCIe slots, so many vendors didn't (and still don't) test their drivers—even if compiled for ARM—on any real hardware.

I hope this changes, especially since Macs now have ARM64, the Pi CM4 supports it, and some other inexpensive SBCs have 4x or even 8x lanes on the more expensive ones (is there an ARM motherboard out there with a 16x lane bus?).

For the Pi in particular, until the CM4 came out, very few people would ever get access to the PCIe bus on the regular Pi 4 model B, so only enough work was done on the kernel side to ensure the VL805 USB controller worked, and a few bits were just never fully implemented or tested (e.g. BAR allocations, MSI-X support, some memory access functions).

Have you talked to broadcom about the PCIe IO space problem? Most other RISCs handle IO space by just mapping into an MMIO space instead. PowerPC, MIPS, and other ARM SoCs have been handling this for decades at this point. It feels weird that their root complex would choke on that, and more likely that there's some unknown config knock you have to setup that hasn't been documented properly. I feel like you might have enough of a publicity reach to get an actual answer on this one.

Quite some recent controllers outright don't implement it, and newer devices don't require it...

Newer GPUs still require it.

> there an ARM motherboard out there with a 16x lane bus?

For servers at sky-high prices only.

For $699, your closest choice might be the NVIDIA Jetson AGX Xavier... PCIe 4.0 but with a x8 slot, and an NVMe M.2 x4 PCIe 4.0 slot too.

SolidRun HoneyComb has about the same slots and price, but instead of weird nvidia closedness you get first-class upstream EDK2 support, everything is about plug & play with ACPI.

The HoneyComb is at PCIe 3.0. Sadly it's at ARMv8.0 (w/ Cortex-A72) so that removes it from consideration totally for me.

And the Xavier has an official-but-still-beta EDK2 firmware available too, which includes ACPI support and can even boot Windows. (https://developer.nvidia.com/embedded/downloads#?search=UEFI...)

Yes, GPU 3D acceleration is currently not supported there yet, but the LX2K doesn't have a GPU at all anyway...

You mentioned work on the kernel side. Is this entirely a hardware issue or are the Linux pcie drivers for the Pi also lacking functionality?

It's a mix, and from my work and discussions on getting a GPU running (or not...), it seems to make debugging even harder if I can't figure out if it's bad memory access or an actual hardware feature that's just not there.

I've been tracking progress on this site / repo: https://pipci.jeffgeerling.com/

You can, with properly written drivers and an open OS.

GNU/Linux obviously is amd64 centric. There's still a tremendous amount of dismissiveness towards anything that's not amd64 (we're even seeing it towards 32 bit x86). So if something doesn't work properly on a non-amd64 system, it's usually not considered a high priority.

The BSDs are much better about this. If something doesn't work, that means it's broken even if it works where 98% of people use it, so it gets fixed. I can, and have, plugged LSI and other RAID cards in to several different non-x86, non-amd64 systems and had them work exactly as expected.

Yeah, I partially wondered because NetBSD specifically has docs talking about ex. plugging "PC" cards into a "Mac" motherboard and being quite emphatic that it should always work because drivers were portable and orthogonal. Sad if Linux isn't doing as well:\

I would guess the peripherals of the processor, bugs in the hardware and firmware etc.

That and the drivers are still murky blobs that I would guess on average don't have much thought put into them from a pure software engineering perspective

I'm curious if we'll see a PCIe Raspberry 4 this March.

If not I'll use Intel Atoms with SATA as backup to my Raspberry 2/4 cluster.

Another option than the mentioned CM4 module might be some RK3399 based SBC, such as the Rockpro64, which comes with a PCIe x4 interface (the Rockpro64 has a open PCIe slot so you could put in a x16 card too).

If you just want to hook up some M.2 NVMe SSD there are also other SBCs with the same chip such as the nanopc-t4 that come with a M.2 slot.

I wish something cheap-ish had more than a single M2, along with a PCIe.

I'm starting to realize you probably want SATA that can be powered independently.

I think I might not wait for Raspberry 4 with PCINVM...

Peripheral Component Interconnect Non-Volatile Memory; mouth full right there... express? tcho tcho! after espresso or just fast?

Why not get a CM4 with the IO Board? PCIe on the regular Raspberry Pi seems too niche for them to consider.

Because I make clusters: https://camo.githubusercontent.com/af3dccb802848a5f72d41c962...

My most recent one will have 6x raspberries with really nice heatsinks for the 4: http://move.rupy.se/file/pi_4.jpg

The cost of the CM and the ergonomics of the design makes this more expensive than going full Intel Atom!

And the Gflops/watt is exactly the same.

Are there any decent cases for the CM4 with the I/O board? Last time I checked there weren't any.

Not yet. I'm working on a couple designs that I'll 3D print for a couple of my own projects.

It's still hard to find both the IO Board and CM4 in stock though :(

that storage pod looks really interesting. Standalone disk enclosure connected to a RAID card. Are there any like these broadly available (in that form factor)?

Reply on news.ycombinator.com