el_bhs

the customers spending billions on the big cloud providers are entitled to much more rigorous and detailed reporting (incl historical data) for end-user-visible performance... I would much rather see the graphs in the article than a green check box (especially when things aren't actually very "green").

(disclaimer: I am one of the people who started opentracing)

Someone sent me this HN thread – nice to see the questions here!

OpenTracing has had a big end-of-2017 with support landing for various service mesh(es), bindings announced from some major vendors (newrelic and datadog), many community contributions in the "core" OpenTracing repos as well as integrations with numerous external projects, lots of uptake from companies large and small, etc.

Anyway, if people have questions, the best place to go is the public Gitter: https://gitter.im/opentracing/public

People are friendly there :)

Also feel free to reach out to me directly. I will probably do a bad job following this thread... bhs@gmail.com

"OpenTracing-compatible" is strict API compatibility in any supported language. The cross-language spec is "terminology-based" since it's, well, cross-language.

There is an open issue about Envoy/linkerd/Istio support here: https://github.com/opentracing/specification/issues/86 (as well as in a number of other locations)

As an OpenTracing contributor, the core value prop still seems quite strong in that instrumentation of OSS dependencies is a massive pain point and should not be tracing-system-specific since it doesn't need to be. There is also value in common protocols and formats, and in that spirit there is interest in broadening scope to include those... though from seeing many companies adopt tracing tech, I haven't observed protocol compatibility as the main pain point or blocker.

what a lovely design!

I don't think that doing manual memory management doubles dev time for experienced devs, no... I just mean that, if you're trying to eliminate GC hiccups by, say, writing a custom allocator in C++ (i.e., exactly what we had to do with this project I was on at Google), it just adds up.

I.e., it's not the manual memory management that's expensive per se, it's that manual memory management opens up optimization paths that, while worthwhile given an appropriately latency-sensitive system, take a long time to walk.

This is a lazy answer, but it needs to be said: it depends on the context and the number of roundtrips to the cache.

If we're talking about a vanilla object cache, I think that it probably only costs 50% or so to make the extra copies and system calls. However, an in-memory "cache" can be something more than just a key-value store. E.g., for the thing I'm building right now, there are some more structurally complex graph structures that need to be traversed with every request to the "cache", and of course making a localhost RPC call for each entry/exit to the cache is really problematic. Though I admit that most caching is unsophisticated stuff, and in those cases it's just a ~2x difference.

It really did take twitter many years to rearchitect and break their most problematic dependencies on Rails, yes... my understanding is that it was a 4-year process.

For my own project, the GC tradeoff with Go (or Java) is acceptable given the relative ease of development w.r.t. C++. Since there are better structures in place to explicitly control the layout of memory, you can do things with freepools, etc, that take pressure of GC.

For high-performance things like the systems I had to build at Google, I don't know how to make things work in the high percentiles without bringing explicit memory management into the picture. Although it makes me feel like a hax0r to talk about doing work like that, the reality is that it adds 50% to dev time, and I think Go/Clojure/Scala/Java are an acceptable compromise in the meantime.

It is possible to build things that minimize GC churn in python/ruby/etc, of course; I don't want to imply that I'm claiming otherwise. But the GC ends up being slower in practice for any number of reasons. I'm not sure if this is true in javascript anymore, actually... it'd be good to get measurements for that, I bet it's improved a lot since javascript VMs have received so much attention in recent years.

Final point: regardless of the language, splitting services out behind clean protobuf/thrift/etc APIs is advantageous for lots of obvious reasons, but one of them is that, when one realizes that sub-service X is the memory hog, one can reimplement that one service in C++ (or similar) without touching anything else. And I guess that's my fantasy for how things will play out for my own stuff. Ask me how it went in a couple of years :)

Sure.

So, for any black-box service endpoint, the latency for any given request is obviously just the time it takes for that operation to complete. Ideally one measures both end-to-end latency from the client and server-side latency in order to understand the impact of the network and, for high-throughput applications, any kernel buffering that takes place.

All of that is obvious, I imagine. By "high-percentile latency", I'm referring to percentiles of a distribution of all latency measurements gathered from a given endpoint over some period of time. If you imagine that distribution as a frequency histogram, the horizontal axis ends up being buckets of latency ranges (e.g., 0-10ms, 10-20ms, 20-30ms, etc), and the bars themselves of course represent the number of samples in each such bucket. What we want to do is determine which bucket contains the 95th percentile (or 99th, or 99.9th) latency value.

You can see such a latency distribution on page 10 of this paper which I published while at Google:

  http://research.google.com/pubs/pub36356.html

Anyway, it is a mouthful to explain latency percentiles, but in practice it ends up being an extremely useful measurement. Average latency is just not that important in interactive applications (webapps or otherwise): what you should be measuring is outlier latency. Every service you've ever heard of at google has pagers set to track high-percentile latency over the trailing 1m or 5m or 10m (etc) for user-facing endpoints.

Coming back to Rails: latency is of course a concern through the entire stack. The reason Rails is so problematic (in my experience) is that people writing gems never seem to realize when they can and should be doing things in parallel, with the possible exception of carefully crafted SQL queries that get parallelized in the database. The Node.js community is a little better in that they don't block on all function calls by convention like folks do in Rails, but it's really all just a "cultural" thing. I don't know off the top of my head how things generally work in Django...

One final thing: GC is a nightmare for high-percentile latency, and any dynamic language has to contend with it. Especially if multiple requests are processed concurrently, which is of course necessary to get reasonable throughput.

Hope this helps.

I can't resist (this is the OP): you are missing the point. It's not just throughput, it's high-percentile latency. Latency is critical if you have 1 billion users or 100 users, and it is difficult to bring the high percentiles of the latency distribution down into reasonable territory on Rails since, by default, all operations are essentially serialized.

JRuby is of course incompatible with C extensions to Rails; the place I used Rails had such dependencies, and so JRuby was not an option. I agree that JRuby is otherwise preferable to cRuby, though.

I'd be curious to hear how Square has found ruby (independent of rails) to be from a maintenance standpoint.

I don't think anyone is arguing that companies fail to grow because of these languages. It's merely that they would grow more quickly once at scale if they didn't have to spend several years rearchitecting while adding few innovations in the meantime (this is what happened at Twitter, for example).

(I wrote TFA -- didn't expect it up on HN!)

I agree that the post would be more compelling if I wrote a benchmark to demonstrate how much faster an in-memory cache is than an off-process or off-machine cache.

From first principles, though, I believe it should be obvious (yes?) that the ~300 nanoseconds it takes to grab a read lock and read from main memory is going to beat the ~1000000 nanoseconds it takes to get a response back from a remote cache over the network. Inasmuch as an application blocks on such cache reads, these sorts of things add up to troublesome latency numbers (and Rails – or at least dallistore – does indeed block on reads like these).

JRuby was off the table for the place I used rails due to reliance on some C extensions.

And I'm sorry if the argument seemed arrogant: I was being tongue-in-cheek about the "unrelenting negativity" part. My point about objectivity was that I tried not to rely on my opinions as much as demonstrable statements. That said, I didn't take the time to actually demonstrate most of those, and for that, shame on me.

Ah, I hadn't seen that, thanks!

For what it's worth, "Go" as a language is not really implicated in that, it's more like the `go` cmdline suite that was causing trouble. I would also contend that the devs were being foolish to do what they did... Assuming everything was in a git repo, the toolchain makes proper use of submodules, and to my mind this sounds like a case of developers fundamentally misunderstanding git, not Go per se.

But my "railing on Rails" (and, to a lesser extend, Node, Django, etc) will not focus on Go... it's more of a general critique about the lifecycle of large software projects written in dynamic languages.

Exactly. With proper discipline and true 100% test coverage, dynamic languages work well. But over time, that ideal is a challenge for most software organizations to actually live by. Or that's been my experience... it's one of those theory vs. practice situations.

@dvt, this is the OP here.

I was hoping that those who already agree with me about dynamic languages would come to understand that Go is different in this respect. I did a lousy (i.e., nonexistent) job making a case to those who don't agree with me [yet! :)] about dynamic languages, though.

I will write a followup post later this week about the long-term maintenance problems associated with languages in the python/ruby/javascript family. I don't think they're "bad" (I was known to advocate for python in certain situations when I was at Google), but they're often inappropriate, and it is my sense that many developers haven't had the requisite large-dynamic-language-project trauma yet to understand that from firsthand experience. (The toughest part about those traumas is that they happen so late in a project's lifecycle that there's no quick way back to safety...)

So I will try to make that case in a future post. Thanks for your thoughts.