WookieRushing

Its basically a compute engine that maintains all state in memory and does distributed computations similarly to Spark.

The big thing it adds is that it isn't stuck to any storage format. It has a connector that lets you load data into it from basically anything, from mysql dbs to hdfs files to whatever. So you can do cross database joins and just not care about where the data lives. You can also output to almost any database too.

RPC systems come in many different levels like programming languages. While there are low level RPC systems which are just a simple layer around web requests, there are others that do a lot more. They can do retries, host selection, and stateful operations like the article mentions.

These systems tend to take care of a bunch of easy to mess up logic which tends to be accumulate around any system that wants to send something and not mess up. Any sufficiently old web request system tends to look identical to a high level RPC system designed badly.

So choosing an RPC system should give you all the features you’d eventually end up building around using web requests without spending your time rewriting it.

Having their own file format could really help them out as it lets them be sole reader and writer of it. File formats for warehouses are generally a little out of date in terms of performance because it requires all the compute engines to be able to read them and some will lag behind others in updating.

The metadata caching helps a ton I’m sure too. When you have to issue the same get file handle on lots of nodes instead of just say one then you lose out a lot on tight latencies and can cause lots of problems for the underlying storage system

The benefits of using googles internal tools isn’t what this paper is mainly about though. They do benefit from it but much more of the paper is explaining how extremely careful caching and having a deep control over the file format allows them to do things insanely fast.

Reading from colossus is only done if it misses the data cache and the data cache has a 90% hit rate. So in effect they’re getting in memory speeds instead of needing to always hit storage.

Otherwise there wouldnt be too much of a difference versus dremel.