Hacker News Reader

Comment by jaegerpicker | Hacker News Reader

While I'm pretty sure that Go is much faster than python in nearly all tasks, I'm still skeptical that a) they built a full replacement of the system in only a week and b) that they didn't improve the algorithm in the go code. I work at a python/django shop and I recently updated a process that could only handle 5000 transactions in 5 minutes to be able to handle over 100,000 in just under 3 minutes. This was all in python would go been a lot faster? Sure but how much is the new code path and how much is the advantage of golang?

View on HN

Also what to point out that I fully realize that Go is a lot faster than python at preforming the same task but this kind of analysis is dangerous because it leads inexperienced engineers to believe that changing languages is a good performance optimization which should usually be the last step/idea considered. It's expensive and dangerous to switch languages mid stream and while it's sometimes the exact cure it needs to be weighed very heavily to make sure it's the right situation. This article, at least the headline misses that piece of the puzzle pretty fully IMO.

Not sure I agree with you there. IMO, porting between languages with similar abstraction levels is not that hard. I ported some Python code, whose logic took weeks of fine-tuning, to C# in less than a day. Go still has a garbage collector and its type system is similar in philosophy to Python's duck typing. Unless you are porting extra dynamic code (eval(), monkey-patching) or deeply intertwined with an unusual library, it should be a fast process. On the other hand, I was leaving Python get real OS threads, not to increase performance.

This obviously also depends heavily on the complexity of the application and the available resources.

Even so, a production-ready transition in a few weeks time doesn't leave a ton of time for benchmarking and testing, but as someone who operates on a fly-by-the-seat ethos more often than I should I'm in no position to be tossing stones around.

This is nitpicky, but Python threads are kernel threads. It's the GIL that restricts CPython to a single core.

OTOH goroutines are green threads multiplexed onto kernel threads (aka N:M thread model).

There has been articles talking about rewritting being less of a terrible idea if your writing a SaaS. Not to mention the throw one out, you will anyway idea. I'd be interested though, in hearing more about what they rewrote into Go. Its most probable its not all of their codebase.

Yes, it sounds like its a websocket server. A lot of that stuff is relatively low complexity if you glue everything together with a message queue since they can be made language agnostic very easily.

Just our realtime system, and we run some asynchronous tasks with Go now, specifying to replace a few things that were done by Celery previously.

See this shitty thing I wrote to bridge the gap: https://github.com/mattrobenolt/go-celery :)

Actually, I think "Go is faster" has two primary built-in points:

1. It's faster in terms of real performance - given its proximity to C, optimization at the compiler level for multiple platforms, that its highly concurrent and run compiled (usually), this is no shocker.

But, 2. It's faster than transitioning from a dynamic scripting language than it would be to transition to, say, C or C++ or Java, etc.

Not that it's syntactically similar (obviously), but that language, testing and deployment design won't present any egregious bottlenecks and that there's so much built into the core library that complements an http-based service such as this.

In other words, the prospect of going from Python (or Rails or PHP or whatever) to "faster language X" has historically imposed an inherent cost in terms of development that I think a language like Go mitigates to some degree by design.

> and run compiled (usually)

I think you are talking about 'go run', which one might easily mistake for an interpreter because of the speed of the compiler, but it's actually compiling your code and running it. Were you talking about something else?

I wasn't talking about go run, no - there have been a few examples of running go interpreted I've seen posted here. They may well just be experiments but I left the "usually" as my caveat.

Edit: I swear there was an interpreter demo'd, but I may well have been thinking of something like agora, which interprets Go-like code within Go.

I was actually fishing for cool stuff I hadn't heard of, like agora. I don't get to dive into very much Go news! Thanks

> I'm still skeptical that a) they built a full replacement of the system in only a week

Keep in mind that Highscalability bases it's articles on other 3rd party research. They don't talk to the people they are writing about, even when those people (like me) offer to help them get their facts right.

In all likelihood it's just completely wrong.

I assure you, I built this in a week. :)

See, a first person source! :)

Nonetheless, my statement otherwise still stands.

While I do certainly make mistakes I take great pains to be as accurate as I can. And I do have a lot of original interviews on the site, though some problems have kept me from doing them lately. I give the sources directly in the article and they are directly from the people involved in the project. And if there are any inaccuracies in any article I would be happy to make corrections. I don't recall when you offered to correct mistakes, but if you did so and I missed it then I apologize.

It seems as if they replaced one component of the system in a week, not the whole thing.

It sounds like this is a very contained server process that does one thing: pull things from rabbitmq and pushes that to nginx which does the push stream. So this could be written in a week easily.

To be fair, there were some optimizations that were made as well. Everyone does that, right? :)

But overall, specifically our realtime service is a hybrid of CPU intensive tasks + lots of network IO. gevent was handling the network io without an issue, but at higher contention, the CPU was choking everything. Switching over to Go removed that contention for us, which was the primary issue that we were seeing.

Right and that's awesome and ALOT more useful than the article because that seems an ideal use case to switch to go over python.

This specific use case is multiplexing messages to lots of connections, which is almost the ideal use case of Go anyways. It wasn't a matter of re-implementing complex business logic. 1 week is reasonable given the specific problem.

> b) that they didn't improve the algorithm in the go code

The port to GO will do it (I think), because it have built-in facilities to do concurrent programing across all the library..

Reply on news.ycombinator.com