They already have a huge impact on performance and in my experience most people don't care which filesystem they use as long as the performance is good.
Some of us care about not losing metadata randomly too... or suddenly having hardlinks get duplicated...
- Apple made APFS, which seems to show roughly the same performance as HFS+ on SSD hardware. Some benchmarks show faster, some show slower, but it's certainly no 5X improvement across the board.
- btrfs and ZFS have some amazing features, but remain niche and don't seem to yet have wide deployment. Linux distros seem locked on EXT4.
- Windows has NTFS (and ReFS for the server)
At a consumer level, I care about: a) reliability b) performance c) power. SSD manufacturers so far have delivered hugely on all three. I'd love to see how much more they could deliver by moving a lot of filesystem functionality onto the device. Sure the first few iterations might suck like early SSDs did, but in a few years?
BTRFS, NTFS, ZFS, EXT4, and APFS/HSF+ don't really matter for general users sure, but each has certain advantages and limitations that make the flexibility should you need it very useful.
I want my storage hardware to have a higher level API than "block device". I don't want to format a drive in some OS-specific format, I want a built-in format that I can access from anything. I want this format to be durable, well specced so that I can easily access historical data. I want embedded devices to support richer storage functionality than afforded by FAT32+.
Consider that before S3, we stored files on servers in bespoke ways where we had to worry about permissions and filesystem differences (path too long!) and encryption and durability and a hundred things that we now... don't. Instead we have a simple API with dozens of conforming backend providers, including the ability to do it yourself. Imagine if computer storage experienced a similar revolution?
Sure, someone can make super-limited drive which only does 3 basic ops (read, "upsert", list). Would it be useful for general purpose computing? I doubt it.
I'm not sure I'd agree with that:
- "no attribute": S3 does actually have quite sophisticated ACLs and permissions that can be set at the object level
- "no way to say "create file but do not overwrite it": there are a few ways you can do that. Versioning, bucket policies, etc.
In fact I disagree with both yourself and the GP with your assumption that S3 is simple. If you want to do anything "enterprisey" with S3 then you absolutely do need to be aware of the maximum size of allowed objects (these days it's 5TB but it wasn't so long ago when it was only around 5GB). Then there's bandwidth costs (which means you need to also take into account VPC traffic and thus S3 endpoints) and ACLs (soooo many different places you can set permissions on objects - it's easy for someone untrained to get totally lost). There's different storage options in S3 too - depending on the frequency you want to request the data.
S3 is only superficially simple - but then the same can be said for any file system. If you just want somewhere to dump files then any fs will work. But when you start needing to performance tune and do all the other considerations that the GP highlighted with local server storage, then S3 quickly becomes as complicated as every other option out there.
This is the most "apples to oranges" comparison I've ever seen!
Filesystem do something totally different than hard disks or SSDs, and you can't replace one with another. One stores data in hardware form, the others defines the format, reliability (e.g. journaling), querying, metadata, attributes, security, etc of the data.
- Currently, SSD controller firmware goes to incredible lengths to present something that is not a contiguous block device (due to the way flash works) as a contiguous block device to the OS. Likely this includes a huge amount of logic for storing and updating the mapping between logical blocks and actual hardware storage location. Since the control APIs currently are very low level, the controller basically has to buffer everything, transform that into what the flash needs to do, and then execute, not unlike how a CPU decodes instructions and converts to uops or reorders. Presumably this is quite difficult as early controllers had all sorts of problems such as performance tanking when the disk got close to full.
- The filesystem then pretends that the very non-uniform storage provided by the SSD is actually a uniform block and introduces a layer of metadata in a bespoke format to determine the actual disk locations of files, directories, and their associated metadata. Along with that, it can include lots of very nice extra features such as encryption, journaling, hashing (for integrity and de-duplication), CoW, symlinks, hardlinks, directory hardlinks, etc.
- It seems likely then that were the storage API to be raised to a higher level (files and metadata, rather than block storage), the SSD controller can implement every one of those features with superior reliability, performance, and power usage. We've already seen where software encryption went to hardware encryption -- the result is that the encryption is mostly transparent to performance. What would happen if a file's extents mapping was simply the physical location mapping the SSD controller is already doing? If if could transparently perform hashing on read/write or keep a proper journal of file operations? Handle CoW transparently?
And all of that implemented in a way that those just become standard storage features rather than requiring a certain OS to run.
Right now, if my filesystem dies, I have a ton of tools and manuals for recovery. This is the format is well known, and diagnostics tools are easily available.
If my SSD dies, it's just gone. In two cases I had, the drive just was not allowing any file reads at all -- I had zero chance to recover the data. The drive internals are totally opaque.
So no, I would prefer my SSDs to be super dumb -- I do not trust the manufacturers to get recovery tools right.
I'd love to see something like Open Channel SSD's ( http://lightnvm.io/ , https://openchannelssd.readthedocs.io/en/latest/ ) becoming mainstream.
Spec at http://lightnvm.io/docs/OCSSD-2_0-20180129.pdf
> This specification defines a class of SSDs named Open-Channel SSDs. They are different from traditional SSDs in that they expose the internal parallelism of an SSD and allow the host to manage them. Maintaining the parallelism on the host enables benefits such as I/O isolation, predictable latencies, and software-defined non-volatile memory management, that scales to hundreds or thousands of workers per terabyte.
Yeah but are you really sure that hardware encryption is safer? Are you sure that it's not secretly saving the key somewhere where it can be extracted? Are you sure that the hardware encryption works at all and that it's not just writing the data unencrypted?
It just that instead of running on main cpu, it would run on SSD controller's microchip. And will probably be binary only, that people wont be able to inspect or fix.
Samsung has good hardware, but not that great software.
I'd rather have the opposite. SSD hardware exposing their internals, to os developers.