Hacker News Reader

> Now lets look at flash. Most people think of flash as a disk replacement because it connects to the computer through a disk interface (even though it doesn't have disk mechanicals). So it seems like a really fast disk. My assertion though is that it isn't a fast disk, its slow memory. It is 'near line' memory.

Not trying to be a party pooper, but flash does not behave like random access memory at all. With RAM, you can erase/overwrite memory at any address at will. With both NAND and NOR flash, you can only update 1s to 0s or 'reset' bits to 1. But this is only done in chunks. Flash is actually called 'flash' because in the early prototypes, there was a visible flash of light when a block was reset to all 1s. This means that to erase part of a block, any data that is to preserved within this block first has to be moved away. So the controller uses a block mapper to keep the address presented on the interface the same, while actually relocating data to someplace else (a bit like the remapping that common hard drives do when a sector goes bad). To further complicate matters, there's only so many times a block can be erased — erasing causes wear. So the block mapper also incorporates a mechanism for wear leveling. This is also why you want to align your partitions to match the erasure blocks. Otherwise you'll get write amplification; you'd write to 1 block of your filesystem, but if that block straddles erasure blocks, it would cause 2 relocations.

There are filesystems in the Linux kernel that are built to do wear leveling (i think it was yaffs/ubifs), they operate on MTD devices (/dev/mtd?) that do not do their own wear leveling. If you use CF or SD cards, or USB flash drives, they'll have a controller which does the wear leveling (but they don't always do this in a sane way).

So, sure, you could get Flash to behave like RAM, but it would be through emulation, by hiding the relocation and wear leveling logic. You could propose doing this in some super-heavy special MMU and I am with you in sofar as that it could be of some benefit to eliminate some abstraction layers like ATA and avoid hitting a bus (pci, or usb on pci).

Well, flash doesn't behave like non-volatile memory, but it is like random access memory. You don't have to wait for a spinning disk to come around in order to read a sector. And, SSDs can read any block at the same speed. HDs read sequential blocks faster. The mechanics of writing/erasing doesn't change the fact that it is much closer to slow memory than fast disk. You can leave the implementation details (wear leveling, etc) to the controllers.

It actually reminds me a lot of old core memory (http://en.wikipedia.org/wiki/Magnetic-core_memory). In order to read core memory, you had to actually try to write to it. If the voltage changed, you guessed wrong, so you then knew the correct value.

> The 'driver' for flash attached as memory is var = ((var type )(0xsome64bitaddress);

It's a nice idea, but the driver for flash needs to do much more than this. For example, blocks need to be erased before they can be re-written, and since they have a limited lifecycle blocks must also be copied around and remapped.

http://en.wikipedia.org/wiki/Wear_leveling

True, if you look at Intel's flash chips they do this internally, you've got the flash 'controller' which sits astride a bunch of actual flash being written in parallel and its doing the wear leveling. I've played a bit with Intel's PCIe attached flash and can vouch for it being quite a bit faster than going through the SATA port, now if we can get them to move it to the FSB so that I can talk to it via the L1/L2/L3 cache then I'll be really happy.

That's an interesting concept but there would still be a need for the much faster and non-volatile DRAM. At which point you need to discern which is which and decide for each data structure where you want to locate it.

The latency of NAND flash will be so much more than DRAM that you'd have to use DRAM as the cache layer before the flash in which case you could just use one of those block caching schemes coupled with a PCIe SSD to get everything that is good with very little trouble at the architecture level.

Absolutely, there will always be a place for DRAM (in bulk 'fast' memory) and for Static RAM (in caches), this is just a new (for this generation) tier beyond DRAM.

I doubt that you could use NAND Flash as RAM substitute as it is right now. Basically it's a block device with page size about 8KB and erase block ~ 2MB. Data could be read and written only in pages, in-chip page load times are on the order of 100us, page write 1ms, block erase -- 5ms, and it takes about 100us to transfer page data in either direction. So, random read/write latencies are 10^2 / 10^1 times lower than HDD's, but still about 10^3 / 10^4 times larger than DRAM's. I guess internal architecture could be redisigned to reduce block sizes and allow parallel operations inside a sigle chip, but it's unclear how much it will affect density and price.

But it would be useful to put NAND controller closer to CPU, remove block device emulation layer and let OS do wear-leveling and block allocation. We could get high-performance swap or put there specialized MTD filesystems like UBIFS.

Not familiar with 'drum' memory I take it? Back in the way back time a bunch of read/write heads all sitting long a drum which was spinning underneath them. It had some similarities to flash, no seek time, you just picked the head to read or write, if you wrote you wrote the entire track because there wasn't an index (usually). Cool stuff.

Your proposed solution will slightly improve latency, not bit rate.

>You want to make some sort of logical computation based on the state of a 50GB data structure, and all you do is follow the pointers? The 'driver' for flash attached as memory is

> var = ((var type )(0xsome64bitaddress);

You don't need any changes in hardware for this, you can use memory mapped files. Main reason that today software serialise everything is interoperability. For example, old versions of Microsoft Office applications did this, modern don't.

Also read about PhantomOS.

> You don't need any changes in hardware for this, you can use memory mapped files.

That still triggers the OS drivers on the other end which has to go through the same complicated drivers to read blocks into RAM before mapping them into the process, so it's nowhere comparable to what he's proposing.

Flash on the main bus would mean the only thing a read would go through would be the MMU page translation, then straight to either the flash chips or the flash controller, without being bottlenecked by passing through other parts of the system.

And that bottleneck is already real - I have SSD setups at works that are pretty much maxing out the PCIe bus on those servers, and they're not even that expensive (as long as you want performance, not large amounts of storage).

Now, there are still challenges, not least that either you're still talking to a flash controller rather than directly to the flash chips, or you have to handle wear levelling etc. yourself if you want to do writes. Both have disadvantages.

Some embedded systems do talk straight to the flash chips themselves and handle the wear levelling directly - I once put Linux on a BIOS-less x86 system with flash directly on the main bus where the "bootloader" did the equivalent of a memcpy to copy the kernel into a suitable location, set up some very basic stuff and jumped to it, and where the "disk" used an FTL (flash translation layer) driver that handled wear levelling in the kernel code.

You get some extra benefits too: You get rid of the split between memory and filesystem address space. In fact, you can kill off the traditional filesystem altogether. I'm excited by that, filesystems are a dirty kludge.

But this would require completely changing how modern operating systems work. No doubt applications developers would love it though.

do you mean filesystems as the hierarchical way to organize stuff or the filesystem implementation that ensures that data remains persisted also in the event of unflushed writes in case of crash/poweroff etc ?

Both, but much more the second.

disks and filesystems as a weird, separate appendage hanging off the side of a computer are conceptually a weak idea.

Once you've unified your address space - stable storage is in the same space as unstable storage, then your entire world changes. The trick then becomes, how do you represent objects (Hint: Others have already tread this ground, see AS/400).

Although many of the problems (i.e. seek time) of rotating disks are overcome by solid state persistent memory, you still need special datastructures to ensure that:

a) data is buffered during write and that a power failure won't leave the storage in an inconsistent state b) blocks to be overwritten are reallocated and erased in background (you cannot simply overwrite a flash block before erasing it) c) the used datastructures (trees etc) are optimized for the size of the flash block d) gracefully handle block corruption

Since it should be possible to remap blocks (either because of (b) or corruption (d)) it's common to have a logical block remapping layer somewhere in the (disk) controller. At this point classical filesystem algorithms and datastructure can easily solve the remaining problems (consistency with journaling and block optimized access like on rotating disks).

Of course, there is another approach for solving (b): see http://en.wikipedia.org/wiki/Log-structured_file_system (actually, log structured filesystem technique can be used to implement the block remapping within the ssd compatibility layer which lies inside the disk controller itself, but that's a detail) or copy on write filesystems like zfs/btrfs etc (although AFAIK they rely on a disk interface which is compatible to a rotating disk).

That's to say that solid state storage is not really just persistent ram where you can apply the same datastructures and procedures you would use in core.

Future technologies might be different and trim down these differences so that you could really have huge amount of super fast word addressable random memory access, where nothing can be lost even in case of power failure. But, even assuming that this might be possible, it's possible that later there will be yet another technology with even better storage density/ cost but which will introduce back some of the complications. So, it's understandable why you want to keep some abstractions, so that you can easily replace the underlying technologies and get most of the benefits.