Seagate Creates an NVMe Hard Disk Drive
pcmag.com
pcmag.com
We used to do IDE emulation over SATA for a while, then we got AHCI over SATA and AHCI over other fabrics. Makes sense to stop carrying all the legacy loads all the time. For people that really need it for compatibility reasons we still have the vortex86 style solutions that generally fit the bill as well as integrated controllers that do PCIe-to-PCI bridging with a classic IDE controller attached to that converted PCI bus. Options will stay, but cutting legacy by default makes sense to me. Except UART of course, that can stay forever.
Edit: I stand corrected, AHCI (like the name implies: Advanced Host Controller Interface) is for the communication up to the controller. Essentially the ATA commands continue to be sent to the drive but the difference is between commanding the controller over IDE or AHCI modes. This is also why the controller then needs to know about the drive, where NVMe doesn't need to know about the drive because the ATA protocol is no longer needed to be understood by the controller (as posted here elsewhere).
AHCI is the native SATA mode, AFAIK explicit mentions of AHCI are mostly over non-SATA interfaces (usually m.2, because an m.2 SSD can be SATA, AHCI over PCIe, or NVMe).
You can have fully standard SATA communication happening between a SATA drive and a SATA-compatible SAS HBA that uses a proprietary non-AHCI driver interface, and from the drive's end of the SATA cable this situation is completely indistinguishable from using a normal SATA HBA.
Likewise, you can have AHCI communication to a PCIe SSD and the OS will think it's talking to a single-port SATA HBA, with the peculiarity that sustained transfers in excess of 6Gbps are possible.
Informally, a lot of BIOSes (another informality there!) would give you the choice back in the day between IDE or AHCI when it came to communicating with the drive.
I think this is where most of the confusion came from.
(Later, a third option appeared for proprietary RAID modes that required vendor-specific drivers, and that eventually led to more insanity and user misunderstandings when NVMe came onto the scene.)
No, I'm no expert in this at all, but every motherboard I've had has had an AHCI/IDE mode toggle for its SATA ports.
(Maybe the same thing is sometimes framed as 'legacy' mode vs. not or something? Just not that I've seen.)
SATA is still IDE, just with different physical layer.
Same for 3.5mm TRS for analog connections :D
There's not much new for analog, which is probably the way it should be.
Here's a reference design from TI showing how one might implement it on a device where you want to be able to switch between using your USB-C connector for data devices and for analog audio devices [1].
USB-C headphones might disagree with you, as the other comment has pointed out (USB-C supports analog audio on data pins in order to create the one-connector-to-rule-them-all).
And as noted in the other comments, this is about simplifying support for things like connectors, LEDs, expanders, and software in data centers where they're mixing NVMe SSDs and HDDs today. It's just simpler to have fewer parts and software differences.
Another advantage I can see is that the overhead to controlling NVMe is somewhat lower than SATA/AHCI, you can use kernel-bypass (like SPDK) to control many drives more efficiently than through the kernel interface, spending less time in the CPU when you control large number of drives from a single server. The HDD bandwidth is rather low so you can easily put a large number of bridges in there to increase the drive count.
But as you say, there's a ton of things that NVMe just does way better though. From the start, it had much more optimized/slimmer & less chatty protocols (mentioned), multiple queues, and much bigger possible queue depths. NVMe was a huge leap forward. And it continues to evolve really good capabilities. Prioritization, streams features, multi-zoned devices... there's so so so much better capabilities available on NVMe.
Of course theres a long history of "multiple interface" drives which are always ugly hacks that turn up as rare collectors items and examples of boondoggle.
Y'know that eventually we'll be running everything off Intel's Optane non-volatile RAM: we won't have block-addressable storage anymore, everything will be directly byte-addressable. All of the storage abstractions that have popped-up over the past decades (tracks, heads, cylinders, ew, block-addressing, unnecessarily large block sizes, etc) will be obsolete because we'll already have perfect-storage.
It's not quite DASD, but it's much better.
"true" Optane hasn't taken over the scene because, to my knowledge, there's no commercially supported OS that's built around a unified memory model (heck, why not have a single memory-space?) for both storage and memory.
We can't even write software that can just reanimate itself from a process image that would be automagically persisted in the unified storage model. We've got a long way to go before we'll see operating systems and applications that take advantage of it.
(Of course, getting those chunks down to cache line sizes does open up a lot of possibilities...)
Also, RAM "block size" is 64B, while for flash its more like 4kB. CPU cache will "deblock" the 64B blocks, but it can't efficiently do it for 4kB ones.
And then there's the speed. Does replacing PCIe with a memory bus actually make a performance difference that's measurable given flash latency?
Also known as a bank. The transfer itself is 64 bytes at a time (burst length 8 for DDR4 x 64 bit channel). But internally, the RAS (row address strobe) is the access of note.
[0]: Zen3 does it for sure, but I think I also saw it somewhere else.
"row" buffer locality is relatively important for energy efficiency, and can account for about a 2x factor in the DRAM's power consumption. Each rank has 4 bank groups of 4 banks each, with the frequency of "activate" commands to banks in the same group being restricted more severely than to separate bank groups. A rank is just a set of chips who's command/address lines are in parallel, and who's data lines are combined to get up to the typical 64/72 bits of a channel. A single chip is just 4, 8, or 16bit wide. A channel can have typically between IIRC 1 and 12 ranks, which are selected with separate chip select lines the controller uses to select which rank the contents of the command/address bus are meant for.
Also, "activate" and "precharge" take about as long (most JEDEC standard timings have literally the same number is cycles for these 3 timings (also known as the "primary" timings)) as the delay between selecting the "column" in a "row" buffer, and the corresponding data bits flowing over the data bus (for reads and writes there may iirc be a one-cycle difference due to sequencing and data buffers between the DDR data bus and the serdes that adapts the nominal 8-long bursts at DDR speeds to parallel accesses to the "row" buffer).
With JEDEC timings for e.g. DDR4-3200AA being 22-22-22 CL-tRCD-tRP ("column" address-to-data delay; "row"-to-"column" address delay; "precharge"-to-"activate" delay), and the burst length being 4 DDR cycles, this is far from random access.
In fact, within a bank, and even neglecting limits on activate frequency (as I'm too lazy to figure out if you can hit them when using just a single bank), with assumed-infinite many "rows" and thus zero "row" locality for random accesses, as well as perfect pipelining and ignoring the relatively rare "precharge"/"activate" pairs at the end of a 1024-columns-wide "row":
Streaming accesses would take 4 cycles per read/write of a 64B cacheline (assuming a 64bit (72 with normal SECDED ECC) DIMM; is less bits, the cacheline would be narrower), archiving 3200 Mbit/s per data pin. Random accesses would take 22+22+22=66 cycles per read/write, archiving 193 ³¹/₃₃ Mbit/s per data pin.
So random accesses are just 6 ²/₃₃ % efficient in the worst case and assuming simplified pathological conditions. In practice, you have 16 banks to schedule your request queue over, and often have multiple ranks (AFAIK typically one rank per 4/8/16GiB on client (max 4 per channel on client, 2 optimal, with 1 having not enough banks and 3-4 limiting clock speeds due to limited transmitter power) and low-capacity servers; one rank per 16-32GiB on high-capacity servers).
There is a slight delay penalty for switching between ranks, iirc most notable when reading from multiple ones directly in sequence or possibly when reading from one and subsequently writing to another (both are pretty bad, but I don't recall which one typically has more stall cycles between transmissions; iirc it's like around 3 cycles (=6 bits due to DDR; close to one wasted burst)).
Wait, are the underlying hardware access size for memory bigger than the default Linux page size (4kb)? Wouldn't that introduce needless inefficiencies?
Can false sharing happen between different pages if they happen to be in the same row?
In practice, the only thing DDR4 banks do is make prefetching an important strategy, thus making sequential performance for DDR4 incredible.
A fact already known to high performance programmers. Accessing byte 65 after byte 64 is much more efficient than accessing byte 9001.
https://faculty-web.msoe.edu/johnsontimoj/EE4980/files4980/m...
diagrams on pages 4-7.
Doesn't OS/400 work that way? (Of course then there is the question to which degree "commercially" should imply "readily open to third-party software and hardware vendors")
> IBM's design of the single-level storage was originally conceived and pioneered by Frank Soltis and Glenn Henry in the late 1970s as a way to build a transitional implementation to computers with 100% solid state memory. The thinking at the time was that disk drives would become obsolete, and would be replaced entirely with some form of solid state memory.
...so they were literally 50-60 (maybe 70-80 at this rate) years ahead of their time.
Suffice it to say that it was promising while it was under development, but Intel neglected to factor in just how much flash vendors would improve performance and reduce cost per bit.
https://www.rambus.com/memory-and-interfaces/server-dimm-chi...
There's a lot of simplifications you can use knowing everything is a world of 512 byte/8k blocks.
Everything that's a linked list of blocks would suddenly be lists of "start and offset" instead, because you might have a file was fragmented as "1Mb at offset A, 32 bytes at offset B, and 5Mb at offset C". You'd likely need some tradeoffs for minimum fragment sizes to avoid a file sharding into a million 32-byte fragments and requiring a huge number of distinct reads to load, and maximum sizes that would be manageable without overloading the recipient device's buffer.
Rumors seem to be that Oxide Computers is trying to make an OS to control many of these but I suspect the end result will be treating the case as a heterogeneous cluster. It's possible that these will continue acting cooperatively rather than anyone being under the misapprehension that they're actually 'in charge', but perhaps a different division of labor will give us some cool benefits.
Would still love to see SQLite or Postgres running directly on a storage subsystem.
I'm not sure what Oxide is doing, but creating an OS to replace the firmware on all kinds of 3rd party devices sounds, well, quite quixotic. I'd guess it's more like a common OS for the firmware of their own hardware (rack controllers and whatnot).
You could say the same about device drivers in Linux. It was a very long process that required many, many hands. It took a long time even to cover all of the common devices let alone the more obscure ones.
Isn't it the interface that is faster and not the protocol? PCIe vs SATA
Edit: after reading more, this article is littered with inaccuracies
The low-level advantages of NVMe 1.0 were mostly about reducing overhead and improving scalability in ways that were not strictly necessary when dealing with mechanical storage and were not possible without breaking compatibility with old storage interfaces. Nothing about eg. the command submission and completion queue structures inherently favor SSDs over hard drives, except that allowing multiple queues per drive each supporting queue lengths of hundreds or thousands of commands is a bit silly in the context of a single hard drive (because you never actually want the OS to enqueue 18 hours worth of IO at once).
As a thought experiment, I think there are usecases for this kind of thing for a hard drive.
The very nature of a hard drive is that sometimes accessing certain data happens to be very cheap - for example, if the head just happens to pass over a block of data on the way to another block of data I asked to read. In that case, the first read was 'free'.
If the drive API could represent this, then very low priority operations, like reading and compressing dormant data, defragmentation, error checking existing data, rebuilding RAID arrays etc. might benefit from such a long queue. Pretty much, a super long queue of "read this data only if you can do so without delaying the actual high priority queue".
But the big problem with having the drive's queue contain a full second or more worth of work (let alone the hours possible with NVMe at hard drive speeds) is that you start needing the ability to cancel or re-order/re-prioritize commands that have already been sent to the drive, unless you're working in an environment with absolutely no QoS targets whatsoever. The drive is the right place for scheduling IO at the millisecond scale, but over longer time horizons it's better to leave things to the OS, which may be able to fulfill a request using a different drive in the array, or provide some feedback/backpressure to the application, or simply have more memory available for buffering and combining operations.
HDD manufacturers get to keep selling their same tech with a different interface. From an end user perspective, a drive like this lets you buy future proof server equipment with the newer interfaces. You can make the plunge to full SSDs once the market's providing what you need.
There are several NVMe features intended to expose some information about the underlying segmentation and allocation of the storage media, for the sake of QoS. Since current multi-actuator drives are merely multiple actuators per spindle but still only one head per platter, this split could be exposed as separate namespaces or separate NVM sets or separate endurance groups. If we ever see multiple heads per platter come back (a la Conner Chinook), that would be best abstracted with multiple queues.
Another complication is of course that the non-PEG slots on the (non-HEDT) platforms are usually electrically only x4 or x1, so bifurcation really only makes sense in the PEG.
Smaller PCIe switches and motherboard chipsets usually support link widths from x4 down to x1. Treating each lane individually goes hand in hand with the fact that many of the lanes provided by a motherboard chipset can be reconfigured between some combination of PCIe, SATA and USB: they design a multi-purpose PHY, put down one or two dozen copies of it at the perimeter of the die, and connect them to an appropriate variety of MACs.
So on the consumer platforms that give you two PEGs best you could do while still having a GPU is stick that riser in the second PEG and use the x8/x8 split. Now the question becomes whether the UEFI allows you to use the x8/x8 bifurcation meant for dual GPU or similar use in an x8/(x4+x4) triple bifurcation kind of setup.
Realistically this entire thing just doesn't make a lot of sense on the consumer platforms because they just don't have enough PCIe lanes out of the CPU. Intel used to be slightly worse here with 20 (of which 4 are reserved for the PCH but you know that), while AM4 has 28 (4 for the I/O hub again). On an HEDT platform with 40+ lanes though...
(When I say bifurcation I meant bifurcation of a slot on the mainboard, not the various ways the slots and ports on the board itself can be configured, though that's technically bifurcation as well (or even switching protocols)
True; but given how PCIe speeds are no longer stalled, we may soon see motherboards offering an x4 slot that can be operated as x1x1x1x1. Currently, the only risers you're likely to find that split a slot into independent x1 ports are intended for crypto mining, and they require switches. A passive (or retimer-only) quad-M.2 riser that only provides one PCIe lane per drive currently sounds a bit bandwidth-starved and wouldn't work with current motherboards. But given PCIe gen5 SSDs or widespread availability of PCIe-native hard drives, those uses for an x4 slot will start to make sense.
Recent Intel chipsets were locked to Intel SSDs and only some magic hardware DRM handshake unlocked bifurcation https://www.asus.com/us/support/FAQ/1037507/ But apparently AMD competition forced their hand and at lest some got unlocked https://forum.level1techs.com/t/new-bios-update-unlocks-4x-p...
On AMD front there doesnt seem to be any problems, plenty of desktop boards with unlocked bifurcation BIOS menus.
Going multi-GPU on a high-end GPU today lets you run a AAA game at 4K with all the eye candy at 150 fps rather than 110 fps, i.e. visually undistinguishable. And at the low end, it's always better performance per dollar to buy a single higher-end card.
Of course this is all orthogonal to your point of PCIe switches. Anyone thinking about multi-GPU will buy a motherboard with at least 2 PCIe 4.0 x16 slots anyway, no switches required. You can even get motherboards with 2 PCIe 5.0 x16 slots without breaking the bank.
I'm pretty sure you're referring to motherboards that have two slots which are mechanically x16, but actually using the second slot borrows half of the lanes from the first slot, leaving you operating at x8+x8. This is not the same as having a full 48+ lane PCIe packet switch between the CPU and two slots each wired for x16, which was very common for high-end consumer motherboards in the PCIe gen2 era (eg. with NVIDIA's NF200 switch).
These days, you can get 32+ lanes straight from the CPU to enable two x16 slots, but that requires stepping up to a bigger CPU socket than the mainstream consumer CPU+motherboard platforms use. And I don't think you can yet buy a motherboard with two PCIe gen5 x16 slots at any price: as far as I can tell, none of the Intel Alder Lake motherboards announced so far include PCIe switches, because such switches have been cost-prohibitive for several generations now.
Also the Power10 systems from IBM hit GA very recently, but I won't expect those to be easy to get (in addition to their price).
If you want a battery backed write cache with flash-based NVMe, get an enterprise drive with power loss protection, and you are golden. You don't generally need the write cache to get reasonable throughput, but you can save a few microseconds in write latency.
I suspect that NVMe HDDs will have the same amount of write cache that existing SATA/SAS HDDs have. If you feel you need a RAID card for RAID and/or cache with SAS or SATA HDDs, you will probably have the same need for a RAID card with NVMe HDDs.
Both Intel and AMD support limited hardware RAID for NVMe natively on some of their CPUs. I don't think this has any of the cache benefits you normally have with a RAID card. That is, it may work well with flash-based NVMe drives, particularly those with power loss protection.
But really, if I'm using hard drives, I'd probably use ZFS with flash-based read and write cache or Storage Spaces with a flash tier and its integrated write cache on flash.
"Limited" here means the motherboard firmware understands the same array format as their proprietary software RAID drivers, so you can boot an OS from the RAID volume.
NVMe-OF has options in form of InfiniBand and ROCE HBA chips that can be configured to act as PCI-E root complexes bifurcated over many many links (in place of default x16 endpoint) with minimal management logic or built-in SoC features. So a network JBOD with NVMe-OF can be essentially one chip + PCI-E links to disks + support/glue logic.
Admittedly, I totally did not know NVMe’s were becoming a thing until a year ago, as I had been in the laptop-only space for a while or didnt need to optimize storage speed when connecting an existing drive to a secondhand motherboard.
I like being ahead of the curve and am now curious whats next
Others correct me if I am wrong.
NVMe in itself is an interface specification, people often use the term NVMe when they are meant M2, the Connector.
You wont get a new connector in the pipeline. But the M.2 is essentially just 4 lane PCI-E express. So every time you get PCI-E express update, currently 4.0, 5.0 around the corner and 6.0 in final draft. 7.0 possibly within next 3-4 years. So you can expect 14GB/s SSD soon, 28GB/s in ~2024, 50GB/s within this decade assuming we could somehow get the SSD controller power usage down to a reasonable level.
So the NVMe card I added to an old motherboard’s PCI-E slot is really just PCI-E on PCI-E? yo dawg
Assuming that's a PCIe to M.2 adapter card, it's just rearranging the wires to a more compact connector. There's no nesting or layering of any protocols. Electrically, nothing changed about how the PCIe signals are carried (though M.2 lacks the 12V power supply), and either way you have NVMe commands encapsulated in PCIe packets (exactly analogous to how your network may have IP packets encapsulated in Ethernet frames).
M.2 is a bit more complex[1] than that, sadly.
[1]: https://en.wikipedia.org/wiki/M.2#Form_factors_and_keying
Traditional PCIe add-in cards (PCIe CEM connector) are still around and also not going to be disappearing anytime soon, but are in decline as many use cases have switched over to other connectors and form factors, particularly for the sake of better hot-swap support.
M.2 (primarily SSDs) is in rapid decline in the server space. It may hang on for a while for boot drives, but for everything else you want hot-swap and better power delivery (12V rather than 3.3V).
The up and coming connector is SFF-TA-1002, used in the EDSFF form factors and a few other applications like the latest OCP NIC form factor. Its smaller configurations are only a bit larger than M.2, and the wider versions are quite a bit denser than regular PCIe add-in card slots. EDSFF provides a variety of form factors suitable for 1U and 2U servers, replacing 2.5" drives. The SFF-TA-1002 connector can also be used as direct replacement for the PCIe CEM connector, but I'm not sure if that's actually going to happen anytime soon.
I haven't seen any sign that EDSFF or SFF-TA-1002 will be showing up in consumer systems. Existing solutions like M.2 and PCIe CEM are good enough for now and the foreseeable future. The older connectors sometimes need to have tolerances tightened up to support higher signal rates, but so far a backwards-incompatible redesign hasn't been necessary to support newer generations of PCIe (though the practical distances usable without retimers have been decreasing).
Are they all repackaged from those 3?
So that explains why only 3 companies make HDDs.
https://en.wikipedia.org/wiki/Seagate_Barracuda#Firmware_bug
https://en.wikipedia.org/wiki/Seagate_Barracuda#Firmware_bug...
https://en.wikipedia.org/wiki/Seagate_Barracuda#Firmware_bug...
https://en.wikipedia.org/wiki/Seagate_Barracuda#Firmware_bug...
https://news.ycombinator.com/item?id=27419342
...and then there's this one which is infamous enough to have its own Wikipedia page:
https://news.ycombinator.com/item?id=27419072
I don't think I've seen something this bad since the IBM DeathStars.
Also, firmware mistake = disk disappears from the bus is pretty common in enterprise SSD land. HPE did it twice, I've seen it on Intel drives, and I'm sure there are others I'm not remembering. Maybe Seagate was trying to be more enterprise... and at least their drives only failed to start up IIRC, if you were running them continuously it was fine (unlike the SSDs)
"Special drivers"? What year is it? Is "the datacenter" run on Windows ME?
Seriously, PCMAG, try a little harder.
The only mention of drivers I can find in the article is an entirely correct reference to the fact that SAS and SATA HBAs frequently require proprietary (or at least vendor-specific) drivers. I'm not aware of any 8+ port SATA HBAs that use the vendor-neutral standard AHCI interface.