Comment by repolfx | Hacker News Reader

Comment by repolfx | Hacker News Reader

Parent Full thread

He meant "centralised in the serialisation library". I don't see how that conflicts with versioned evolution, especially because protobufs historically did verify the presence of fields.

The story behind "required considered harmful" at Google is really quite shameful. I was there at the time and couldn't quite believe people were making that argument. Beyond all the logical problems with it, they were basically saying Jeff Dean didn't know what he was doing when he made that decision (except they never actually said it explicitly because the notion itself would have been subjected to ridicule).

I used to like protobufs but I wouldn't use proto3. It seems to have regressed over time instead of getting better. Partly this is because the wire format is not evolvable. Protobufs do not have a "this is protobufs v1" header in them. This also explains the type system issues: Google cannot/will not evolve the wire format so each new version of protobufs is just different tooling and APIs over it.

I was there at the time too and couldn't believe people were defending required. I had already run into enough problems with it that I stopped using it earlier, and I worked on a relatively small system with not that many moving parts! I can hardly imagine how much pain required must have caused in something like the indexing pipeline.

The thing I realized was that even if a field is present, you almost always still want to do further checks on it before declaring the whole thing valid and good to go. "required" just tells you that it's an integer, for example, it doesn't tell you if it's in the range [0, 1000). Or that a string is a valid filesystem path. Or that two parallel arrays are the same size. You have to do those things anyway, so the has_... check is no big deal. And making everything optional is really nice when you have to make large changes to your protocol and don't want to have to carry around dummy values forever.

Protocol buffers weren't designed from scratch by Jeff Dean, they evolved from various ad-hoc ways of specifying query parameters (which, ok, were probably mostly written by Jeff), so I don't think it reflects badly on him to say that required was a mistake. Groups were a mistake in retrospect too, but no one thinks that means Jeff is an idiot.

Btw, I realize this argument is very similar to the argument for dynamic typing + lots of unit tests instead of static typing: "you have to write the tests anyway, and the tests will check the types".

I'd like to note that even if you're a static typing fan and don't believe that argument for code, you can still believe it for serialization and protocols, because data schemas and protocols evolve in different ways than code does (they're almost duals of each other).

The argument from Jeff Dean authority suffers from the fact that no matter how smart you are, getting some trade-offs right will require empirical validation.

Just having multiple end-points or doing version negotiation might work fine in some cases, but I can't see a particular good way field-replacement evolution would work with exclusively required fields in general,

What's your suggested approach?

Reply on news.ycombinator.com