Hacker News Reader

What is the overhead (roughly) for this guy? I do absolutely love the idea of using elasticsearch for this instead of a traditional database.

Using celery or django-rq to delay the processing also seems reasonable.

View on HN

Tbh, probably fairly bad atm. I haven't done anything empirical but atm Silk will create a record for every SQL query executed during the request/response cycle effectively doubling the number of queries. Not only that but it also saves down any non-binary HTTP body to a TextField, and this isn't yet configurable.

I think the Elasticsearch/Celery option is a good shout if this were to ever be used in production. Celery would help with the issue of response time but it wouldn't solve the load impact on the database (although I guess could also configure a different SQL database in Django).

I will certainly do some investigation into this at some point :)

StatsD might be better since you are just recording datapoints + text field I'm assuming.

https://pypi.python.org/pypi/python-statsd

Sounds interesting... i'll take a look. The fact that its UDP would be helpful in avoiding the need for a dependency on celery/message queues.

What's it like in terms of getting the data back out?

You can write your plugin for the back end to dump it. It defaults to graphite which would be familiar enough I think.

Reply on news.ycombinator.com