Why SAS and not NVMe? You can buy all NVMe enclosures/servers now.
Is this solely aimed at replacing drives in existing enclosures? Not much good there as most of them will be SAS6. They'll still work, but not nearly as fast as SAS12 nor NVMe.
Is this solely aimed at replacing drives in existing enclosures? Not much good there as most of them will be SAS6. They'll still work, but not nearly as fast as SAS12 nor NVMe.
Similarly, SAS supports multipath for HA while PCI-E does not appear to offer that ability. If it does, then I have yet to hear of an example of hardware implementing it.
Lastly, the idea is likely to appeal to those that care about density. For that purpose, it is better to lower costs and requiring a multitude of PCI express slots is not a great way of doing that. Motherboards with more PCI-E slots are expensive due to the layers needed to accommodate the circuits required by them. Assuming JBODs for such things exist (using a chip to share lanes among multiple devices), they would be expensive and also add to cost.
There is also a decent write up on this here:
NVMe scales just fine. As for the livelihood of those selling these storage appliances, well check out NetApp stock price lately.
Also,I wouldn't want to run thousands of these SAS drives in today's storage appliances. Even with high performance, the control plane and rebuild/rebalance stuff would be a nightmare. Trying to imagine a 50 node Isilon cluster with these things... not pretty.
As for being different than virtio because it needs PCI-E lanes, what keeps QEMU from emulating however many are needed?
The easiest problems are the ones that go away if you throw money at them and NVMe drastically expands the set of such problems. Most small-mid-sized companies have DBs in the 10Gb-1TB range. If you have a single table that's 100GB in size, you can parse through every single row in just under a minute! This means you can actually use an easy to implement O(n) algorithm instead of trying to make O(1) or O(log n) fit your problem. NVMe SSDs are not that advantageous for companies that are built to scale horizontally on AWS. They are amazing when you have a monolithic DB that you can't partition/shard/cluster easily.
You're forgetting seek latency. It's orders of magnitude better with SSD, but it's still not necessarily zero. Depending on how the data is laid out and queried you can pay the seek cost per row, which multiplied by the number of rows (100GB+) isn't trivial.
Nearly all of the tick (>4GB/day) databases I've used aren't laid row oriented.
Even in the absence of variable-width fields, the presence of nullable fields causes the majority of database tables to have variable-width rows. In any case, neither of these are reasons why common databases do or do not lay rows out sequentially on disk (some do, some don't).
Even if the DB server selectively read columns of each row (none of the common open source SQL databases do), they do so via the OS, which works in terms of pages. Reading a single byte of a page will cause a minimum of 4kb of IO to be made to the disk.
Now, unless you're using a DB server that uses O_DIRECT or POSIX_FADV_RANDOM (I just checked and Postgres doesn't), Linux will aggressively readahead at least (it's tunable) 128kb for any random read by default, so even issuing a one byte read to the kernel, device IO will still only occur in a minimum of 128kb chunks, with the remainder living in the page cache until userspace requests it.
Database servers additionally are very likely to have their own larger-than-a-byte-sized buffers in order to avoid system call latency, so the requests they make are never going to be quite so small.
The logic being that in the days of spinning media, evicting 124kb of cold page cache in favour of avoiding a seek a few microseconds later was definitely worth it (a seek being a ~14ms stall on rotating disks)
This is why I said it was high level, but hopefully illustrated the point. In addition to the disk page size, you also have all the various metadata associated with the file(s). So, reading a byte from a page can imply reading even more data than the block size (4KiB current).
> Now, unless you're using a DB server that uses O_DIRECT or POSIX_FADV_RANDOM (I just checked and Postgres doesn't), Linux will aggressively readahead at least (it's tunable) 128kb for any random read by default, so even issuing a one byte read to the kernel, device IO will still only occur in a minimum of 128kb chunks, with the remainder living in the page cache until userspace requests it.
AFAIK, Linux only reads ahead if it detects a sequential pattern, or if you specify POSIX_FADV_SEQUENTIAL (double normal). But, as far as the query is concerned, all of the data read that isn't necessary is effectively subtracted from the overall throughput.
I was trying to illustrate the importance of seek latency (~80us vs. ~9-14ms), but yes there are a myriad of other concerns when you're trying to maximize disk throughput.
Have you tried to buy one in the past year? If you're not in the Fortune 100, fat effing chance! Here's how it goes, Samsung announces NVMe product, companies beat down their door screaming "take my money!" and Samsung conveniently "cancels" the product for sale. They made it. They just sold their entire production run to a handful of customers. Maybe, just maybe you can get a couple hundred units if you're willing to wait 3-4 months and someone returns some or changes an order after delivery (and you get the returns).
Even second tier (non Intel, non Samsung) suppliers are sold out. About the only thing you can buy now is HGST because no one wanted their stuff in the first place, and they jacked up their prices in response to other vendors' product shortages.
Yes, NVMe is on fire right now. Everyone wants it. I wouldn't put new tech like this into a 3-4 year old system because of a sunk cost fallacy. NVMe is not also exactly "new". They're already on version 1.2 (or 1.3) of the spec. Intel has gone through 2 major NVMe product revisions (with the third out in 3 months).
Also, Samsung isn't exactly breaking 3D NAND ground here. Novachips did it last year, and in a NVMe interface, too.
But, you know, I'm not a fortune 100 company, and this seems to have a reasonable shipping time:
http://www.newegg.com/Product/Product.aspx?Item=N82E16820167...
SFF-8639 (or U.2 as the branding is) is the way forward.