Hacker News Reader

Comment by mattrobenolt | Hacker News Reader

It's definitely not a blanket statement. For us, it is. Considering most requests aren't just 1 simple query either. If you even do 5 simple queries, each at 20ms each, that's 100ms spend just waiting for the db. Plus other systems in place.

And honestly for us, we'd be happy if all queries were ~20ms. :) Lots are unfortunately slower than that. Especially if all 45k/s were doing the queries.

View on HN

With asynchronous (go-like) channels becoming so popular, I'd love to see some non-blocking Django ORM database calling features. That way, you could fire off a few non-blocking db requests and keep executing on the app server (eg. rendering?) until you hit a point where you needed the database data.

I know you could roll your own implementation... but building this into the ORM would still be really slick.

There are frameworks that work like this, iirc. I think some in Scala, but I'm not sure.

IMO, the added complexity wouldn't be worth it with things like Varnish. I think there are better solutions and this is attacking the wrong problem.

But, YRMV. It depends on the situation at hand always.

Basically anything living in the akka+spray world is written this way. But akka+spray is not an framework in the same sense that django is.

Are there async drivers for e.g. mysql yet? Last time I did that I had a very nice async spray frontend, but it was all backed by a pool of regular blocking threads because that was the only way to use the database driver.

There is this, but it's not particularly mature: https://github.com/mauricio/postgresql-async

I also use https://github.com/etaty/rediscala for accessing redis.

But even without async drivers for MySQL, you can still do:

    val sqlResult: Future[StuffFromSql] = future {
      val stmt = connection.createStatement("...")
      ...
    }

This will tie up a thread, but you can still run other actions in parallel with it.

If you do this, execute the future on a different dispatcher so you aren't blocking the default dispatcher.

This can cause problems if you tie up the default dispatcher with blocking SQL queries while using clustering: the cluster gossip won't be able to fire and if your failure detection settings are low, that node can be marked unreachable by the rest of the cluster.

AppEngine NDB does this for their datastore. Even works with Django, though not with the ORM (you have to use AppEngine's datastore API).

How many database servers do you have? How do you have them handle the queue?

1 primary master, with a few read slaves and a few shards.

Thanks for answering my questions. (:

Reply on news.ycombinator.com