mattrobenolt

If you want async, you could really just keep consuming from the channel with goroutines until your hardware catches on fire. There's nothing in Go that is limiting this behavior.

Of course. I'm not saying that they're 100% async, just defending the fact that they're not 100% blocking.

This is true. :) Our definition of "realtime" is slightly different than a CS definition. Sub-1s end to end time usually falls under "realtime" when it comes to the web, but it depends on what the subject is of course. In our case, that's perfectly acceptable. We're obviously shooting for lower, but at this point, we can't do much to improve network latency and other factors involved. So for now... this is good. :)

They can be. By default a `make(chan int)` is blocking, but you can give it a buffer size with `make(chan int, 1000)` or something and it won't block until it's full.

Yes. Our realtime updates vote counts and new comments over websockets if your browser supports it. If not, we fall back to some more typical HTTP streaming.

It definitely wasn't helping. But it was not our bottleneck, I assure you.

The technical barrier for Erlang is much much greater in my opinion than Go. Introducing something into our stack is a very tricky situation. We are mostly all Python engineers, so bringing in something new has to be intuitive for others to pick up on and get involved.

There will be more in the future. :)

Fortunately, it's not as black and white in the real world. Quick and dirty hacks are a real thing that will never go away. If anything, having the ability to do those under stressful scenarios is a good quality to have.

We were using Thoonk.

Jobs were originally pushed from our Django app, into a Celery job, which consumed the message, and published into Thoonk. Thoonk was then consumed by the old Python backend.

With this rewrite, we've also removed Thoonk and are using a fanout exchange in Rabbit directly. Go consumes the Celery job from the Rabbit queue.

Well, the latency referred to is not factor of the Go code. It's more of a factor of our nginx front end which are handling a few million concurrent connections. The time spent doing computation is being measured in nanoseconds.

I was referring to our Python processes being CPU bound. Not Go. :) Yes, I'm familiar with GOMAXPROCS.

Our assumptions were that the GIL was getting in the way with our excessive concurrency. gevent + PyPy is not really a safe thing yet.

This entire project was also stood up in a week, so it's not like this service is some monster service with a lot of work invested into it. It was small enough that instrumenting and figuring out where the problems may lie and figuring them out would have probably taken just as much time.

Most of the latency involves other components in our system. This isn't just computational time. Most of the actual computation with our transformers and whatnot are done in less than 1ms.

Are you referring to the cgo stuff?

This is very true. We're definitely looking at Go for "services", not as something to replace our entire Django app with.

If you're CPU bound, I think it's a good option to at least check out. It can work with real threads, and doesn't compete with the GIL. In our case, our actual CPU was low, but due to the high concurrency, any little bit of CPU work requires a context switch and contends with everything, for a little work. We alleviated that by just using a bunch of processes.

Will it replace all of our Python? Highly doubt it. Some very specific pieces? Probably.

This was partially intention on my part. A lot of the problems that we had with our Python backends were a bit unknown. We had some guesses, and not the right instrumentation to dig into it. There are memory leaks in gevent, and the CPU that was being consumed was ridiculous.

Internally, we had each played with Go for a little bit, and felt very natural to us. There weren't any other considerations really.

For comparison, we're still dark writing to the old realtime just to make sure that nothing is broken, and old realtime is consuming easily 16x resources and technically doing a bit less since it's not really publishing anymore.

Same here. The amount of effort that was required to achieve better results was very minimal. I'm excited to see what I can do when I fully understand Go inside and out to make things even better. :)