Hacker News Reader

"The theory that this market exists in any significant way never pans out in practice."

The transition from HDD to SSD technology is a counter example to your claim. It was a drop-in replacement (same SATA interface) and the tech significantly improved performance without any other software modification needed.

SSDs are orders of magnitude faster than HDD, enough to justify the cost differential and investments to rethink the software stack.

Octane was an incremental improvement in performance when compared to SSD, and in cost when compared to RAM. An awkward spot to be in.

>Octane was an incremental improvement in performance when compared to SSD

Optane PM was byte-addressable and had latency of ~300 ns. It rendered the entire block storage abstraction obsolete. It was so radical that using it effectively would require throwing out the assumptions all OSes have made about IO for the past fifty years.

The fact that it offered an incremental improvement even when its unique capabilities were completely ignored shows how phenomenally capable the technology was.

Didn't SSD's get a start with laptops and mobile devices? There the advantage of being shock resistant is massive and unsolvable with any other solution.

For a superior but new tech to stick, it has to find a viable market to be first self-sustaining, only then can it attack larger and more lucrative markets (Back in those days, PC was probably more lucrative than mobile). Only when SSDs matured enough, did they start eating up the entire storage market.

With SSDs you really feel the performance difference (especially latency) in everyday life. Someone who had experienced an SSD never wanted to go back to a HDD for storing anything but data, since program start-up times are just so much faster when there is no mechanical seeking involved.

The difference between Optane and an SSD was never so striking.

At least for consumer devices, I think the 'primordial soup' spark of motivation was the original lean and mean netbooks (with the initial push to web apps over running full clients locally, before they became low spec windows laptops), with compact flash ATA adapters to displace HDDs. Power consumption was a factor as well IIRC

I wouldn't really say they started with mobile/laptops

They came in 2.5 inch form factor from the very beginning, the laptop shape... which fits pretty much anything. Desktops and servers use it.

The packaging was trivial. They were just cost prohibitive for anything more than an operating system drive.

This worked out to our benefit, the OS is where random access performance shines

It might be fair to say flash used in SSDs started in other applications... but it's important to remember that's also different flash.

It wasn't until the flash changed significantly that we got SSDs (durability), changing everything we once knew. Price, applications, etc.

>Broadly, the notional value of storage technology like Optane is that it offers a drop-in improvement in storage performance without requiring any software modification, albeit at a significant cost premium.

I don't think so at all. The value of Optane was that it was dramatically different, offering byte-addressable storage with latency of hundreds of nanoseconds. Getting the most out of this requires radically redesigning how modern systems store data, including ditching the idea of block storage.

Optane was the fastest traditional SSD on the market by a pretty wide margin, but that's not all it could have been.

>They use the storage so well that the gains by putting something in the middle will be marginal or non-existent, and certainly not worth the extra cost.

I don't agree with this either. How do you propose to get latency down to match Optane using ordinary SSDs? It doesn't matter how much hardware you throw at the problem; parallelism cannot reduce latency. You need another storage system to hit those targets. Maybe that can be a more traditional DRAM cache with battery backup, but that's only adequate in bursty workloads.

Putting "something in the middle" isn't worth it if the rest of the architecture stays the same. But it could bring about tremendous speedups if the new capabilities were used.

It says here latency for Optane read is about 10 µs.[1].

It says here an SLC drive has the same latency for sequential read (random read is 5x higher though). [2]

[1]https://www.intel.com/content/www/us/en/products/docs/memory...

[2]https://www.solidigm.com/products/data-center/d7/p5810.html

What about the 1-10 billion dollar company whose business isn’t technology, like say some mid tier insurance company running MySQL with a combination of off the shelf specialists software and in-house software. They would benefit from faster storage and they can’t tune their stack like Facebook

Obviously it failed as a product but I am not so sure that the cause is as you lay it out