Hacker News Reader

Comment by patrec | Hacker News Reader

> All you've managed to do is decentralize sanity-checking logic from a well-defined boundary and push the responsibility of doing it throughout your entire codebase.

One wonders if the author has practical experience with evolving a complex system composed of multiple independent services without downtime. Centralized sanity checking conflicts with transitioning services stepwise from e.g. foo.a foo.new_a by first deploying producers that create send both foo.a and foo.new_a and then phasing out foo.a after all consumers are updated.

Services which communicate via some general purpose, strongly typed schema language that's still lax enough to evolve backwards-compatibly are far easier to understand, maintain and evolve in my experience than alternatives such as json rest apis, overly tied down static or completely ad hoc protocols.

View on HN

He meant "centralised in the serialisation library". I don't see how that conflicts with versioned evolution, especially because protobufs historically did verify the presence of fields.

The story behind "required considered harmful" at Google is really quite shameful. I was there at the time and couldn't quite believe people were making that argument. Beyond all the logical problems with it, they were basically saying Jeff Dean didn't know what he was doing when he made that decision (except they never actually said it explicitly because the notion itself would have been subjected to ridicule).

I used to like protobufs but I wouldn't use proto3. It seems to have regressed over time instead of getting better. Partly this is because the wire format is not evolvable. Protobufs do not have a "this is protobufs v1" header in them. This also explains the type system issues: Google cannot/will not evolve the wire format so each new version of protobufs is just different tooling and APIs over it.

The argument from Jeff Dean authority suffers from the fact that no matter how smart you are, getting some trade-offs right will require empirical validation.

Just having multiple end-points or doing version negotiation might work fine in some cases, but I can't see a particular good way field-replacement evolution would work with exclusively required fields in general,

What's your suggested approach?

I was there at the time too and couldn't believe people were defending required. I had already run into enough problems with it that I stopped using it earlier, and I worked on a relatively small system with not that many moving parts! I can hardly imagine how much pain required must have caused in something like the indexing pipeline.

The thing I realized was that even if a field is present, you almost always still want to do further checks on it before declaring the whole thing valid and good to go. "required" just tells you that it's an integer, for example, it doesn't tell you if it's in the range [0, 1000). Or that a string is a valid filesystem path. Or that two parallel arrays are the same size. You have to do those things anyway, so the has_... check is no big deal. And making everything optional is really nice when you have to make large changes to your protocol and don't want to have to carry around dummy values forever.

Protocol buffers weren't designed from scratch by Jeff Dean, they evolved from various ad-hoc ways of specifying query parameters (which, ok, were probably mostly written by Jeff), so I don't think it reflects badly on him to say that required was a mistake. Groups were a mistake in retrospect too, but no one thinks that means Jeff is an idiot.

Btw, I realize this argument is very similar to the argument for dynamic typing + lots of unit tests instead of static typing: "you have to write the tests anyway, and the tests will check the types".

I'd like to note that even if you're a static typing fan and don't believe that argument for code, you can still believe it for serialization and protocols, because data schemas and protocols evolve in different ways than code does (they're almost duals of each other).

Reply on news.ycombinator.com