Ask HN: Quick 'n dirty Django scaling tips? (Wikileaks just tweeted and fb'd us) | Hacker News Reader

Ask HN: Quick 'n dirty Django scaling tips? (Wikileaks just tweeted and fb'd us)

10 pointswhalesalad5 comments

Hey HN,

The site in question: http://epicstep.com

I recently helped launch a startup called Epic Step that crowdsources billboards in America. I work at a dev shop in Los Angeles, we built it, but it's not technically ours. The guys decided to launch it in a somewhat limited alpha mode (good idea) to prevent the site from imploding. Basically, launch with a few campaigns and let people interact with those, limiting the creation of new ones.

One of the first campaigns is for a Wikileaks billboard in Los Angeles. Wikileaks tweeted about it today, and just threw it on their Facebook page. The Facebook story has ~700 likes and a lot of comments, and the site is getting hit pretty good right now.

The site is built in Django. The stack it's running on right now is a 512mb Slicehost with nginx and uWSGI, MySQL for the db. It's actually doing pretty damn well right now... server was swapping but has about 130mb of free memory right now. CPU usage is also surprisingly low, most of what we're facing right now is high mem usage.

Right now my battle plan is: if mem usage gets too incredibly high, just resize the slice and once things settle out we'll built out better infrastructure.

TLDR: Are there any other things we can do to improve performance for this evening? I'm worried that the Wikileaks traffic is going to destroy this!

I'm terrified because while two very high-traffic sources are linking to us, the site is doing fine right now and that just doesn't sit right. Either the crowd hasn't really come yet, or I actually did a decent job with the server and I need to just relax.

- Get more memory for sure.

- Try using memcached first. Eg. libraries like johhny-cache.

- Add varnish proxy in front.

- You can cache full pages, with the django cache middleware. Commonly most pages are same for everyone not logged in. Even if you're logged in you can cache full pages and just pull specific items via a jquery scatter plugin for example. I do that to pull the "welcome user, login/logout" bits on top.

if you need more help, email me.

Excellent workshop on the subject of deployment and scaling of django apps by Jacob Kaplan-Moss

http://ontwik.com/python/django-deployment-workshop-by-jacob...

Nothing really specific to Django you can do. I guess you can disable USE_I18N and USE_I18N settings if you're not using them and disable middleware and context processors you're not using.

Other then that:

- Cache everything you can. (orm/query cache, template cache, proxy cache)

- Optimize queries you can't cache. (use django-debug-toolbar, django-devserver, or enable slow query logging in mysql to view them)

- If you have lots of joins, denormalize your model fields.

- Use queues and cron jobs for long running processes.

- Offload media to another server like s3.

- Add ram.

>the site is getting hit pretty good right now.

and

>Either the crowd hasn't really come yet, or I actually did a decent job with the server

Isn't it more or less a proven fact that traffic levels are generally lower during the weekends? With this in mind, I think you should definitely expect - and be prepared for - at least somewhat higher traffic come Monday.

On the other hand, as you said, current traffic is also fairly heavy and the stack is holding good so it may continue to hold up with higher traffic too.

I'd add just one point to what sidmitra and amccloud have suggested:

- check/tune MySQL config too (although in our experience, this usually affects cpu usage more than it does memory, it may be worth giving it a look.)

HTH

For the absolute clueless about configuring MySQL, there's a nice script that provides some guidelines on what to do based on your server config. Check out "tuning primer" <http://www.day32.com/MySQL/>;

Reply on news.ycombinator.com