mattrobenolt

To be fair, there were some optimizations that were made as well. Everyone does that, right? :)

But overall, specifically our realtime service is a hybrid of CPU intensive tasks + lots of network IO. gevent was handling the network io without an issue, but at higher contention, the CPU was choking everything. Switching over to Go removed that contention for us, which was the primary issue that we were seeing.

We're not using a framework. This is a tiny component and the rest of Disqus is still Django.

Disqus (http://disqus.com/) needs some people to help break computers.

http://boards.greenhouse.io/disqus?t=58y0kn#.UpuMT2RDsoM

Let me know if something doesn't stand out. We're looking specifically for some ops people (possible with an interest in Go) and Django.

It obviously depends on the project, but for something like Disqus or anything else large in terms of size, I'd still go with Django.

For some personal projects, I like to use werkzeug, but they usually are very small and don't do much. :)

I also write things personally in Go if it makes sense. But the Go work is typically not for web services.

Template fragment caching is not a good thing. We don't do this anymore. If you have to cache your template fragments, I generally think you're doing something wrong. Gonna have a bad time.

We run 40 processes on each with 16 cores and 24G of RAM.

http://blog.disqus.com/post/51155103801/trying-out-this-go-t... This has replaced that. ;)

Servers are much cheaper than humans.

First of all, the amount of time that'd be required to rewrite things in said "higher performance" language with the hopes of getting some improvements would be very costly. It's not something that'd be done in even a month by one person. It'd be a whole company effort. Learning curve of new language, etc.

With that aside, I guarantee, for our use case at least, I'd rather invest my time into tuning a Varnish config and save the same # of servers without all the man power wasted.

Our server costs are much much cheaper than our human costs.

We run many more than 12 processes per server. That's how you can get around that. :) But that's frankly more of a Python architecture problem than Django. Django is compatible with coroutine libraries like gevent, just most of the ecosystem is not compatible, so that makes things difficult at the size we're at.

Realistically though, this isn't a problem for us and if things are done right, shouldn't be a problem for you either. Just use more processes and you'll be fine unless you're running out of RAM or something.

FWIW, PostgreSQL is our primary data store. It works well. ;)

Varnish caches dynamic requests.

Obviously you can't cache everything. Hence why we still have 15k/s that are hitting out backends.

There are lots of more complicated things we can do with ESI if we really wanted to get into it.

Sure. Any CDN is better than no CDN. :)

I think what I'd like to get across more is that a lot of our dynamic content can be "static" as well, without going the static files route. Being static for shorter periods of time is something that we can achieve with Varnish, and it's definitely an overlooked concept in the community.

CloudFront is slow. :) Check out Fastly.

Yeah, but that's an older unmaintained version. Official is at https://github.com/lib/pq, and I already shaved my yak. :) https://github.com/lib/pq/pull/135

Fastly is used for some of our traffic yes, but not all. Our main app is not behind Fastly FWIW.

I also write Go probably just as much as Python anymore, and the biggest set backs I have in Go is the immaturity of the ecosystem. Almost any project I work on, I have to shave 10 yaks to get things working right.

My most recent yak was fixing lib/pq so I could query against our production dbs. :)

Also, to touch on your point about cookies, we do a few tricks to working around them and make conscious decisions about stripping them. My talks explain that a bit and how we design our app to work around it.

I appreciate the kind words. :) Of course profiling is important.

Even with starting on a greenfield project of this scale, I don't think I'd choose Go (which I spend a lot of time writing). The ecosystems aren't as mature in libraries and whatnot as Python is.

But aside from that, we do profile, and we know that we have some bottlenecks with actual query time. Not to say that this is the sole contributor, but it plays a role.

And you're right about us being in a position to be able to leverage Varnish to it's fullest. We have a large number of reads vs writes, and a large percentage of those are anonymous users.

Even if we had a faster backend, I don't see why we'd avoid something as simple and useful as Varnish. Even caching things for 1 second is invaluable when doing 45k/s. Even if our backend was some super optimized C code, it's pointless.

Varnish also provides us some other benefits, like being able to serve a "stale" version if we're down and smooth over hiccups.

We do a mix of both, but mostly the actual ORM methods.

This is a very simplified example. We do something similar to avoid a stampede.

1 primary master, with a few read slaves and a few shards.

There are frameworks that work like this, iirc. I think some in Scala, but I'm not sure.

IMO, the added complexity wouldn't be worth it with things like Varnish. I think there are better solutions and this is attacking the wrong problem.

But, YRMV. It depends on the situation at hand always.

We still use the ORM for most things. If you know how to use it right, you can make your queries pretty efficient. I'm not claiming this is the best solution, but it works. It takes patience and understanding to use efficiently and make sure you're not using excessive amounts of queries. Specifically avoiding O(n) queries. Which are easy to achieve in Django.

Another thing to consider is contention and competition for the database's attention. If a query is fast in isolation, it's probably not going to be fast at tens of thousands per second.

We don't run gevent for disqus.com. We do some some background workers and whatnot, but our main app is single threaded with a large number of processes on each machine.

It's definitely not a blanket statement. For us, it is. Considering most requests aren't just 1 simple query either. If you even do 5 simple queries, each at 20ms each, that's 100ms spend just waiting for the db. Plus other systems in place.

And honestly for us, we'd be happy if all queries were ~20ms. :) Lots are unfortunately slower than that. Especially if all 45k/s were doing the queries.

uwsgi

Sure, but the idea is knowing how and when to use your tools properly and what they're good for. It allows us to continue using Django.