Clever RDMA Technique Delivers Distributed Memory Pooling | Hacker News Reader

Clever RDMA Technique Delivers Distributed Memory Pooling

nextplatform.com

36 pointsrbanffy23 comments

As Greg points out this has come up multiple times and in multiple strategies for implementation.

In the early 2000's I lead the design of a memory based Ethernet appliance which we called "Network Accessible Memory". The basic idea combined the fact that FPGAs were pretty cheap, you could easily build a gigabit network interface on one side and a DDR2/3 interface on the other, and with a very simple ethernet protocol have terabytes of memory basically a couple of microseconds away from a CPU as opposed to 'swap' which was milliseconds away at best. (we also simulated the architecture using a machine just exporting memory to the network)

The two biggest wins are that the memory is already 'live' when machines that might use it boot up so they don't spend time reading in state, and in a multi-processor storage subsystem you can put the buffer cache into memory without tying up any individual processor's main memory which gives you faster throughput and lower latency as you're not turning around the disk interfaces for meta data reads and mixed r/w workloads don't blow up your buffer cache.

All very cool. But all very "specialized" too.

Fun -- Texas Memory Systems was already shipping a product in that era that was FPGA-based. It did a variety of complicated network protocols, like Fibre Channel and eventually InfiniBand, with a pure FPGA implementation. The guy who ran the company was also an FPGA wizard. I was on a telecon with him once.

https://en.wikipedia.org/wiki/Texas_Memory_Systems

Yeah, I think they are cited in the patent we did. The key differences were a) no non-volatile storage at all, and b) we took 16 memory 'blades' (1GBit each) and combined them into a single 10gbit backed 'controller'. It leveraged a lot of ideas from the disk shelves at the time. It did give you 1GB/s read/write fully raid protected memory (with equivalent FEC bits in the packets to catch network corruption) to the network.

I expect TMS would have gotten there eventually once they moved on from 'storage' on the network/fabric to 'memory' on the network/fabric.

This is super neat! Comments like this keep me coming back to HN. Thanks for sharing!

Single System Image is some cool tech in this area, where basically you're running a single operating system instance across the network, which sees each computer as just more available computing resources, and you can for example migrate processes from one computer to another.

There are also database systems that use a ring of nodes connected with RDMA and optimize for that data layout, creating a sort of merry-go-round of data: http://homepages.cwi.nl/~mk/datacyclotron.pdf

That is pretty cool, all the dates are place holders, when was it published? Tom Lyon had a startup in this space that was trying to build network accessible memory systems to solve the 'process migration' problem. That is the problem where moving a process on a Linux system takes too long relative to the benefit.

Google looked at some proposals that were kinda sorta like this when they were soliciting ideas for their 'energy proportional computing' effort. Basically they were attacking the problem that the difference between an idle machine and a fully loaded machine was about 25% power usage. That isn't held against your PUE (after all the machine is 'running') but it does mean you could theoretically increase your energy efficiency by 50% if you could leave idle machines powered off and effectively bring them on line quickly enough to meet your design goals.

It seems the paper was published around 2010-2011. This is Martin Kersten, same guy who did MonetDB. I'm not sure if the idea went anywhere since it was very radical and wacky, but perhaps it's waiting for its time to come :-)

The neat thing about it isn't that it /can/ work in such an architecture, it's more that it's optimized to leverage it. Their idea is that you can spin data through the ring one way, and spin requests the other way, and you can have a multi-node distributed data system while avoiding a ton of global coordination.

> you could theoretically increase your energy efficiency by 50% if you could leave idle machines powered off and effectively bring them on line quickly enough to meet your design goals

It's a cool pattern that I keep seeing of people trying similar ideas at different scales! Multicore processors for mobile computers also turn cores on and off to save power, and migrate and balance processes across cores, I believe.

Similarly, the process migration thing could be solved at the language level rather than the OS level, which I think is what some erlang stuff does - as long as you have global coordination and can serialize process state and code, why not?

Google ended up using RDMA. The TIMELY paper, unfortunately, does not mention publicly the applications and services taking advantage of it, but the Tensorflow site confirms that it's one of them.

I also worked in that space several years ago. RDMA-based paging is a very tempting (and sexy) concept. There are however several challenges that make it very impractical in commercial scenarios:

1. Availability: what happens when a node goes down ?

2. Page sharing: how should / can multiple writers be supported ?

3. Kernel overhead: how do you hook in the paging subsystem without significant overhead ?

4. Cost/performance: rare are the companies that would blindly jump into an IB network investment; and RoCE is not as fast as IB (although it may be catching up with 25/40/100 Gb cards).

In the end we ended up designing a RDMA-based DHT. Besides, this kind of approach becomes less relevant nowadays with solutions like Intel Optane.

Awesome, did write ups appear anywhere of what you learned? I would enjoy reading them.

   > 1. Availability: what happens when a node goes down ?

Pretty typically you fail. One of the things we noted was that making a "shelf" that could get 5 9's (99.999% uptime) was a lot easier than a system that got five 9's. In the NAM design the RAM was ECC (bit protection), the individual cards were RAIDed across (card failure protection), dual power supplies and controller (no state that isn't in the memory so you could have active passive controllers with no synchronization protocol) and a very basic optimistic locking protocol (cost free on success, resync on collision which for this use case was pretty reasonable (not a lot of write contention)).

   > 2. Page sharing: how should / can multiple writers
        be supported ?

The architecture was single writer, multiple readers. It was not designed (although could probably be adapted) to prevent other network nodes from reading data or snooping data reads across the network. We talked briefly about using multi-cast in a shoot down protocol to allow machines to invalidate their lock state. The idea was that nobody 'cached' these blocks so invalidating caches wasn't an issue. That said, looking at the way AMD and Intel manage the Layer1 (per core)/Layer2 (across cores) cache layers for their chips had me wondering if that would have been useful here. Generally if you write and your nonce is invalid you re-read the current nonce and re-write with the updated nonce. Also important to note that in our case it was a 'data' cache not an execution cache.

   > 3. Kernel overhead: how do you hook in the paging
        subsystem without significant overhead ?

We were using Ethernet, which had lower latency and overhead than using the (SATA/IDE) layer in Linux. Not IP, Ethernet. So to fetch a 'block' (8K bytes) you'd drop the address of that block on the network pipe and the packet would go out, and the response would come back pretty much instantly. (1 - 2uS).

   > 4. Cost/performance: rare are the companies that 
        would blindly jump into an IB network investment;
        and RoCE is not as fast as IB (although it may
        be catching up with 25/40/100 Gb cards).

Our thought was you got a benefit just using the existing Ethernet network you had. (after all you were mostly getting rid of disk seeks which were milliseconds for Ethernet packets (microseconds) who doesn't like a 1000:1 reduction in latency :-). In our case it was 'add this box to your storage stack and get a xx% boost in throughput or yy% decrease in latency.' Where the numbers could be calculated based on your workload and your read/write mix. If the sale price had 55 points of margin then we had successfully done our job :-).

Here is a publication that describes a more advanced embodiment of the system I mentioned: https://www.computer.org/csdl/proceedings/sc/2015/3723/00/28... ; it don't think it explicitly compares to distributed RDMA-based paging systems but most of its non-RDMA features were implemented to address the shortcomings of PGAS systems (hardware-accelerated or not).

I know this is a "pop" news writeup and not a proper paper, but this same thing has been done many times before, for example the Cray T3E had this sort of thing built into its OS. One of the interesting challenges is that the networking stack allocates memory, so you have to be careful to avoid deadlocks when memory runs low.

A user-level example is the Global Arrays toolkit from PNNL, released in 1994: http://www.emsl.pnl.gov/docs/global_arraysdev/index.shtml

It looks like PNNL has been slow to update their own Global Arrays web page. The source has finally moved to github: https://github.com/GlobalArrays/ga

I noticed this a few months ago when NWChem build scripts added a tool to fetch GA from github. It's nice because although the NWChem svn repo has long enabled read-only public access, the svn repo for GA was closed.

PGAS libraries like GA are rather different from swap, though. Jumbomem <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.158.... might be a better comparison in userspace if you must have that sort of thing.

I didn't see the source code for infiniswap linked in the article. Here it is: https://github.com/Infiniswap/infiniswap

This sounds like a modern follow on in the spirit of work done at UW, Duke and UBC in the 90s called "GMS". The interconnect was ATM at first, and then Myrinet (which is a sort of predecessor to Infiniband). The GMS research was done on DEC Alphas running DEC UNIX (and later FreeBSD) and involved allowing applications transparent access to huge amounts of RAM over the network.

And in fact, it looks like they cite some of the GMS papers..

there was a whole movement exploring distributed shared memory in the early 90s, using VM hardware to trigger distributed swap with coherency protocols on top. in fact there was a parallel version of OSF/1

https://en.wikipedia.org/wiki/Distributed_shared_memory

which isn't to say that low latency and high bandwidth doesn't make this a lot more feasible. fault tolerance is pretty hard though :)

In this case they're writing everything to a local SSD. The other usual techniques are RAID-1 (multiple copies) or a higher RAID level (if you want to conserve memory.)

I think infiniswap is pretty neat technically. The real solution to this problem is to design your application so it doesn't swap though.

If you have a lot of unused memory in your cluster, and your application runs a lot faster if you use it, how is that a bad thing?

Of course very few usage patterns can avoid swap entirely, but in general, avoiding it is faster than hitting it.

Almost all usage patterns can avoid swap. In this case they're intentionally using a ram-network swap device because it's cost-effective... faster than avoiding swap.

HPC applications typically have to avoid swapping, since they are likely to be limited by the resource manager and compute nodes typically have minimal or no swap space anyway. (The reason you might want a big swap area is to be able to suspend jobs in favour of higher-priority ones.) It's often a significant performance issue if you even need to go off the NUMA node to memory.

Reply on news.ycombinator.com