I don't understand the need for the payload server from the slides. That makes me wonder - why not just use a HTTP server to serve the static files (e.g. nginx)? I'm sure I'm missing the obvious, but I'm probably not the only person wondering it.
Surely, one would probably need a separate external tool that would pre-fill (nginx wouldn't know that some file's pending before it's requested) and clean up caches (provided that rules are complex than trivial LRU removal on some threshold), as one probably wouldn't like webserver doing this unsuitable job, but that should be another story.
I think the biggest reason would be that almost any off-the-shelf server software would struggle in a Google server environment. They have solved scale at the machine level, and he alludes to this on one slide:
... why aren't you using the cluster file systems like everybody else?
... cluster file systems own disk time on your machine, not you.
> Does an off-the-shelf httpd support ACLs?
Not really. You inevitably end up writing custom code to conform to your particular requirements and/or existing systems. If you want high-performance, you end up writing it in C as a module for Apache/nginx/whatever.
> How easy would it be to make them support google storage instead of a file system?
Unless said storage system is presented to userspace through ordinary file interfaces, same as above. There's no general turn-key solution built into webservers. The problem space is too wide.
Using Go in this way gets you good performance, simple architecture, maintainability, and easy deployment with total flexibility to do whatever you need in order to solve your version of the problem. There are no straightjackets, you don't have to conform to (or find ways around) anyone else's conception of the problem space.
Then again, I'm not sure why the caching wasn't better handled in the CDN that lives in front of it (cache hierarchies work well) leaving this server to simply serve only the very first request.
I guess that some details on the slides would answer those questions, but if anybody here know the answer (some slides were pretty obscure if you're not already familiar with file serving and/or google architecture).
0) copy all the files from the DFS to the local storage.
1) Attempt to make a distributed filesystem available as a mount point (for example, Hadoop FUSE) on the nginx server so it can serve the data.
2) Make the nginx server know the distributed filesystem API and talk to it via user space directly.
The former leads to insanity (writing a really good, fast FUSE implementation is devlishly hard). The latter can work; I've seen a number of open source codes adapted to distributed filesystems.
This is made easiest if the underlying includes an IO abstraction layer. From a quick skim, it seems nginx has a OS abstraction layer (win32 and unix implementations):, which is more than sufficient.: http://trac.nginx.org/nginx/browser/nginx/src/os/{win32,unix.... From that, it wouldn't be hard to write a "distributed FS variant" (although IO and OS are not orthogonal concepts).
The next problem you're going to have is performance. nginx and other systems really are written with the expectation that IO is local, high throughput, and low-latency. Opening a remote file often takes some time (hundreds of milliseconds), and serial roundtrips cause additional latency, especially if DFS and server are not within 5ms RTT.
The next logical step is to write a readahead layer (if you're serving files larger than a single read block to the DFS) to get better throughput.
A combination of the above techniques, with a bit of tuning and elbow grease, is a good foundation for scalable (in terms of file size, total # of files, etC) serving.