Comment by edanm | Hacker News Reader

Comment by edanm | Hacker News Reader

Parent Full thread

A question I hope someone here can answer:

I'm now working on my first major webapp, so I've been learning how to be a sysadmin, pretty much from scratch (almost no previous experience with running webapps, lots of experience with other things though).

Anyway, I've been trying to find resources to help learn, but it's been extremely difficult. These are the questions that are still bugging me, hopefully someone here can point me to some good reading about them. Note that my stack is Python/Django over Apache (mod_wsgi) right now. Also note that my site has a "ping" system, where each client connected continuously pings the server every few seconds to see if any new info arrived, making me have to handle a larger amount of requests per second, I believe.

The questions:

1. What kind of load do I need to handle? Is 12,000 requests per second terrible/good/great performance? How do I go about figuring out how many people are online at once for most sites? How do I even estimate it?

2. How can I test the performance of my application? I've learned that Apache Bench is used a lot, but are there better tools?

3. What are the best tools to help me monitor and understand the load on my server?

4. How do I go about understanding the bottlenecks in my application? Right now, my Apache process is taking most of the cpu. What does that imply about where I should optimize?

Sorry to braindump, but I've been looking for answers to these questions online and haven't found any clear help.

As a "meta-answer", you might consider breaking these questions up and asking them on one of the stackexchange sites.

However:

1. "What kind of load" depends on your anticipated visits to the site. There's no substitute for real traffic, but simulation can do in a pinch. Likewise there's no substitute for simulation, but capacity planning tools can do in a pinch. Look at The Art of Capacity Planning, The Art of Application Performance Testing and Guerilla Capacity Planning.

2. There are many tools, some quite sophisticated. Look at siege, httperf and others like them. Consider also using automatic UAT tools (like Cucumber) as the basis for performance and stress tests.

3. Look into monit, zenoss or nagios. There are others.

4. Profiling, profiling, profiling. And ditch Apache, it's a memory hog. Consider lighttpd or nginx.

Thanks, I'll look into those.

When you say ditch Apache, do you mean completely? I know a fairly common setup is to use nginx+Apache to serve Django (with nginx serving the static content and Apache the Django code). Should I be running everything from nginx?

Personally I don't see the point in keeping Apache. It just adds more moving parts and more points of failure to have two HTTP daemons.

I'd ditch it entirely and switch to WSGI or FastCGI.

You should at least try nginx/gunicorn (http://gunicorn.org/). And if your clients are constantly pinging and the data hasn't changed, make sure you are caching it in memcache (etc) to relieve the database.

http://ericholscher.com/blog/2010/aug/16/lessons-learned-das...

Honestly, I'm a little scared of using something like gunicorn. I could be wrong, but it seems like an extremely new technology, especially when compared to Apache, which has a lot of history behind it.

You can have a failover, so if one breaks you can quickly boot the apache one back up.

With Django - or any ORM it's a good idea to keep an eye on the SQL that your application is running. Use the django debug toolbar[1] and run your SQL queries against a profiler. Most of the time, that's where your serious bottlenecks will be. If a query happens to be a bottleneck, there are ways the ORM can improve performance (e.g. select_related() to reduce the number of queries in a loop, only() and defer() to reduce the number of columns in a query, and so on[2]. In some cases this may even involve using pure SQL or caching heavily used queries.

Look at using processes vs threads with mod_wsgi, which should help improve performance. Ultimately however you might find nginx a better option, particularly if you are using it for static media/load balancing anyway. I'd suggest uwsgi (also heard good things about gunicorn). Use supervisord or another process monitor to make sure the lights stay on.

[1] https://github.com/robhudson/django-debug-toolbar [2] http://docs.djangoproject.com/en/dev/ref/models/querysets

"Is 12,000 requests per second terrible/good/great performance?"

Re: this, I don't understand this type of benchmark. A 'request' is completely undefined. Serving 12k 5-byte static pages is quite different from serving 12k frontpages with 25 db queries behind them. Is there an industry-standard interpretation of 'request' that I'm not aware of?

if the guy re-formulates his question like

"is 12000 requests the norm for a popular web site"

would it be better

No.

damn.

i'm happy i have no questions! :)

I realize now that my "No." was misplaced, my "No" was the answer to "is 12,000 req/sec normal for a popular website".

Reply on news.ycombinator.com