Hacker News Reader

Comment by botverse | Hacker News Reader

You use it in Python. OP writes the C code for you.

As soon as you do anything else than return 'hello, world' in Python, your performance will drop dramatically. Just parsing a tiny JSON string a million times in python takes 7s (i5-2467M).

  $ time python -c "import json;
  for i in range(0, 1000000):
      json.loads('{\"data\": \"hello, world\"}')"

  real	0m7.055s
  user	0m6.951s
  sys	0m0.063s

It is no use optimising one part of a system if other parts take 99% of the time.

As it stands, the title of the post would be more accurately stated as 'million requests per second with C'.

FWIW, you can use ujson, which is much faster than the builtin stdlib json parser as well. I've seen some larger webapps in python do just this.

json:

    $ time python -c "import json;
    for i in range(0, 1000000):
          json.loads('{\"data\": \"hello, world\"}')"

    real	0m3.142s
    user	0m3.098s
    sys	0m0.034s

ujson

    $ time python -c "import ujson as json;
    for i in range(0, 1000000):
          json.loads('{\"data\": \"hello, world\"}')"

    real	0m0.444s
    user	0m0.399s
    sys	0m0.028s

Like with many things, it's a question of being smart about your choice of libraries.

Also, just because I was curious:

    $ time pypy -c "import json;
    for i in range(0, 1000000):
          json.loads('{\"data\": \"hello, world\"}')"

    real	0m0.422s
    user	0m0.386s
    sys	0m0.032s

If you use a C library and have hardly any Python code running it obviously goes faster. That misses the point I was making. At some point you'll have to write some Python code, and that will lower the number of requests dramatically. If you're talking millions of requests per second, you will have maybe 10s of thousands of CPU cycles per request. That doesn't buy you much in Python. Case in point: parsing a trivial json string in Python eats up your budget.

Of course, most websites aren't dealing with anywhere near that volume of requests, and then Python is fine.

Pypy is interesting, but last time I looked it didn't support C extensions and hence can't be used with the poster's library.

time python -c "import ujson; for i in xrange(0, 1000000): json.loads('{\"data\": \"hello, world\"}')"

0.291s

Well, that was an easy way to reduce 7 seconds to 0.3.

You might want to use the timeit module instead:

  $ python -m timeit --setup 'import json' "json.loads('{\"data\": \"hello, world\"}')"
  100000 loops, best of 3: 9.34 usec per loop

Reply on news.ycombinator.com