jbeda

Check out https://algorithmia.com/. Stuff like this plus a bunch more. Real business model so you can have more confidence in it.

> Facebook and Google don't really do Work From Home and don't have a lot of remote locations for development depending on where they are. I don't know what they offer datacenter staff, but in many areas, that's all they have for "not CA".

This is just untrue. I haven't looked closely at Facebook recently (beyond watching the [office in Seattle](http://www.geekwire.com/2015/sneak-peek-facebook-plans-epic-...) grow quickly) but traditionally Google has had ~50% eng in MTV and the rest spread out. NY is the second largest office but Seattle and Bangalore are pretty big. Seattle (and Kirkland) are now at ~2000 people (https://twitter.com/juberti/status/675022362438316032).

If you want to use a reasonably priced non-Docker-run registry that is available today, check out the Google Container Registry: https://cloud.google.com/container-registry/

It recently hit GA and is super fast. You can use it from anywhere -- not just GCE.

(Disclaimer: I used to work at Google on GCE/GKE/Kubernetes).

When operating at scale, you will, once in a while, have corruption. Even if you use ECC RAM, once in a while you'll have a double bit flip. And it doesn't look like Backblaze uses ECC (https://www.backblaze.com/blog/storage-pod-4-5-tweaking-a-pr...) despite good evidence that ECC is necessary (PDF: http://static.googleusercontent.com/media/research.google.co...). Even if you do have ECC, you'll once in a while have a bad NIC that with HW offload that will corrupt the TCP stream silently.

This is all rare, but it does happen. This is why the GCS team wants to know if you are seeing corruption on file upload as it might be some bad hardware failing in a non-obvious way.

Using a hash or CRC here is totally necessary. Often times CRCs in TCP fail due to corruption outside the network stack. Having an end to end check will catch, say, memory bit flips and such after data comes off the wire.

But there is no call for a cryptographic hash here. This isn't being used as any sort of ID or to verify integrity outside of corruption.

Took a quick look at the API. For context, I was involved in the early days of Google Cloud Storage.

It is surprising that they didn't make it compatible with the S3 API -- at least for common object/bucket create/delete. This will require more code to be written and it will be harder to adapt client libraries.

The API documentation is here: https://www.backblaze.com/b2/docs/

Other notes:

* The lack of scalable front-end load balancing is shown by the fact that they require users to first make an API call to get an upload URL followed by doing the actual upload.

* They require a SHA1 hash when uploading objects. This is probably overkill over a cheaper CRC. In addition, it means that users have to make 2 passes to upload -- first to compute the hash and then another to upload. This can slow uploads of large objects dramatically. A better method is to allow users to omit the hash and return it in the upload response. Then compare that response with a hash computed while uploading. In the rare case that the object was corrupted in transit, delete/retry. GCS docs here: https://cloud.google.com/storage/docs/gsutil/commands/cp#che...

Very good point. I wrote a blog post on this topic to try and characterize different types of lock-in. http://www.eightypercent.net/post/types-of-lock-in.html

This was as also part of the strategy when we decided to open source kubernetes. Having an open alternative made the commercial offering (Google Container Engine -- GKE) much stronger because of the reduced dev lock-in.

[Author here]

I'd love to see a decentralized image naming and distribution system. I'm pushing hard for this to happen as part of the OCI. See https://groups.google.com/a/opencontainers.org/forum/#!topic....

Author here.

Lots of comments on the term "modern" and the fact that this is container based.

While there are obviously other ways to skin this cat, for me, I want the following out of my production stack:

* Self healing and self managing. If a machine fails, I don't want to have to think about it. The system should just work.

* Supports microservices. The idea of breaking your app into smaller components (regardless of the name) can help you to scale your engineering organization by keeping the dev team for each µs small enough that a 2 pizza team can own it.

* Efficient. I want a stack that doesn't require a lot of hand holding to make sure I'm not wasting a ton of resources.

* Debuggable. Complex applications can be hard to debug. Having good strategies for application specific monitoring and log collection/aggregation can really help to provide insights into the stack.

I think that there are disadvantages to using a PaaS. It can bring all of these components together, but it also means that you don't benefit from best of breed systems at each layer.

Also, up until now most PaaSes were vertically integrated and opaque. I imagine that the PaaS of the future will look more like OpenShift where it is a curated set of components in this stack instead of an opaque stack.

Many of the layers defined there are necessary for any well run production system (things like logging and monitoring).

A lot of the extra components allow for dynamic placement and management of workloads. This allows for the system to be self healing and self managed in the face of hardware failure.

In addition, much of this automatic management (once set up) can allow for you to run more different things. It can reduce the operations cost of microservices and let you factor your app in a way where you can scale the development team.

Hopefully, over time, this stack will come pre-tested and pre-packaged. Things look super complex now because of the abundance of options. As things settle, it'll get easier to reap the benefits without taking on the cognitive burden.

Very true this is container based and is very much in the style of what I experienced over 10 years at Google. I put a disclaimer in the post to that.

I was also considering using the term "cloud native" over "modern" but didn't want to bring definition of that term into the discussion.

My bar for modern is that it should be self healing -- hardware/VMs should fail and the system should be able to recover without missing a beat. That requires a level of management and dynamism that is hard to get without containers. (Burning AMIs w/ autoscaling groups is a good alternative).

Looks like 8i is using fewer cameras with a narrower field. The processing doesn't look nearly as clean. The compression doesn't look as good as it didn't download realtime. The MSR video claims that they can deliver their content at 12mbps.

My guess is that the MSR set up is a pretty large array of kinect like cameras. Pretty cool how they clean things up and connect through time to drive compression. Also think it is cool that they detect where hands and faces are and make the mesh more dense there.

But, the 8i folks have a demo on the web and MSR has a pre-canned video.

I haven't seen vagga before. It looks very interesting.

But, since it is still using an OS based package manager (Ubuntu, for example) it doesn't address all of the issues that I outlined. Specifically packages can't be cached across different images by the host. It also isn't obvious which packages are included in which image without cracking the image open.

Thanks for the pointer!

I agree that they are orthogonal but I'm not convinced that they are the right tool for the job. They can do all sorts of things above and beyond what is needed. Those capabilities come at a cost. For instance, they also manage things like ensuring that servers are running and other transient state.

In addition -- I'd consider the "rerun the config tool on a pre-existing container" to be an anti-pattern. This creates more and more layers to the Docker file that have to be downloaded every time that container is run. I'd much rather build each container from a bedrock up with well versioned dependencies (not as easy as it should be with many package managers).

Agreed -- but how do you figure out which packages are in an image without cracking that image?

Quick -- what version of OpenSSL is in the golang Docker images? (https://registry.hub.docker.com/_/golang/). Short of downloading them and poking around the file system, I can't tell.

I think this is really orthogonal. Puppet/Chef/Ansible/Salt all help to configure an environment and work with the native package manager for the OS. What I'm suggesting here is to instead create a new package manager that is native to the container world and is used at build time.

There is still a need for an improved tool set to build the containers and use this package manager. Perhaps these tools could expand into that area.

As for long-running processes -- I'd consider that an anti-pattern. If there is a critical bug, the correct thing to do would be to build a new (versioned) image and restart (either blue/green or a rolling update) to that new image. This also enables rollback if your fix breaks things further.

Thanks for the pointers -- I'll dig in to those to get more ideas.

Also agree that config-as-package is part of this too.

As for statically linked binaries -- this solves some of it but not all. Still hard to figure out which version of openssl is actually running in production. Also falls down in the world of dynamic languages where you app is a bunch of rb/php/py files.

Immutable images are the name of the game. Any sort of surgical update to image A would result in image A-prime.

Any sort of orchestration system will help to run containers and will know what images you are running but not which packages you are running inside those images.

With something like this a cluster admin could find all the containers across a 10k node cluster are running an unpatched version of openssl. Or could block running containers that are using unapproved packages.

"Order dependence" is the fact that Dockerfile is stictly linnear -- even when 2 steps won't interact in any way. It makes caching and reuse more difficult.

I looked closely at Nix and there is a lot to like there. The thing that turned me off is that it is just too complex. Going from something like Dockerfiles to Nix is just a huge leap.

Example: http://sandervanderburg.blogspot.com/2014/07/managing-privat...

A lesson from Docker is that we have to make this dead simple. It has to be something you can grok in 15-60m.

Happy to answer questions and brainstorm ideas here.

I suggest you also look at GCE for network performance. It is something that the Google networking team has been working hard on and, because of the unique network fabric at Google, should scale to lots of VMs.

Also, you can get equivalent "IP per container" using advanced routing with GCE. Flannel has a GCE specific back end where it inserts routes for each subnet assigned to each range. That will eliminate the extra level of encap.

(Disclaimer: I started GCE but am no longer at Google.)

Kubernetes uses Docker (or now rkt) to do the actual containing. The focus is on scheduling and managing lots of containers.

You may run O(10) containers on a single machine. When you run O(10k)+ containers in a cluster you need new tools to manage things.

I remember when we first did this on IE5.5. It fell out naturally from some improvements to the rendering system and a dev (forgot his name! Don?) prototyped it to show off his new subsystem. It eventually got supported because, at the time, we would do anything we could to differentiate.

Long time ago...

I'm late to this thread, but hopefully I can add some value.

I've recently been through this for my own personal email. I have a domain with some interesting rewrites to the address that I forward to GMail. I put together my set up into a Docker container:

https://github.com/jbeda/docker-postfix-forwarder

It is worth breaking this down into each direction -- sending and receiving.

Sending

This is the easy side. You simply run an SMTP server and have GMail use this to send. If you want others to not mark you as spam you'll want to look at setting up SPF (publishing which IPs are allowed to send from your domain) and perhaps DKIM (digitally sign your email).

GMail used to allow sending from aliases directly without an external SMTP server but that is disabled. Old accounts are grandfathered in.

Receiving

This is where things get complicated. Essentially, you are having folks send mail to your server and you are turning around and relaying that email to GMail. GMail has a hard job here. It doesn't know if it should trust you and that you are acting on behalf of the user or if you are an open relay sending spam. I've found after I change my forwarding set up I have to police my spam folder for a couple of days to retrain GMail.

The real complication here is SPF. Sending domains will publish their own SPF records. When doing this they can either specify a soft or hard fail. If they specify a soft fail (`~all`) then there is a chance that GMail won't mark it as spam. But if they use soft fail (`-all`) then it will go to spam all the time. The problem is that the sending domain (say evite.com) doesn't list your relay IP as having permission to send that mail and so GMail respects that.

Having GMail pull the mail via POP is one solution here but that introduces latency.

Or you can rewrite the "envelope sender" so that you are honest about the mail coming through your server. The accepted scheme to do this is SRS. This is not a silver bullet though. If you are forwarding a lot of spam, GMail may decide that your SRS domain is spammy and penalize all incoming mail.

Also, if you are forwarding a lot of spam, GMail will throttle you. It'll have you back off and wait to forward more mail. Your best bet is to find ways to eliminate obvious spam before you forward to GMail.

My solution seems to be working okay for now, but it is a pain the ass and I'm honestly not sure it is worth it.

EDIT: Here is a tutorial that I used to inform my approach. It is worth digging in to. http://seasonofcode.com/posts/setting-up-dkim-and-srs-in-pos...

Actually, this is intentional. Pre-andromeda, we hit ARP caching issues when testing scale for GCE. The model to let the SDN figure out the other end (and stub out things like ARP) was carried forward with Andromeda.

(I'm the original TL for GCE)

Update: the discovery endpoint is pretty simple and now documented: https://github.com/docker/swarm/blob/master/discovery/README...

You are right -- perhaps I'm being unfair. It is Docker's right to run their project as they see fit.

I'm glad to see the new projects (swarm, at least) layered on top of Docker. It is good to see those sink/swim on their own. I was uncomfortable when Docker was (last week) going down the path of building everything into their existing binary.

The advisory board is just that -- advisory. It has no teeth. All approval of anything goes through Docker.com/Solomon.

It is worth reading through the notes from the last/only meeting: https://docs.google.com/document/d/1JfWNzfwptsMgSx82QyWH_Aj0...

The advisory board is just that -- advisory. It has absolutely no teeth. The open source project is 100% owned by Docker, Inc. Solomon believes strongly in a "firewall" between OSS Docker and Docker, Inc -- but it breaks down in various places where it comes to any online service.

Examples:

* The docker hub gets special privileges in the image namespace. It isn't built on DNS but instead rooted in a namespace owned by Docker, Inc.

* The newly announced swarm depends on a discovery endpoint that (as far as I've seen) is closed source and undocumented.

I'd love to see the online parts of the OSS docker project have the same "firewall" as Solomon believe he holds for the OSS components.