Hacker News Reader

Comment by ramses0 | Hacker News Reader

This article is trash. They start with "obvious" python at 12s, run it with pypy instead for 3s, and then rewrite and optimize a D version from 3s to 1s w/o attempting any further optimization of the python version(!?!).

In my opinion omit all of the discussion on python and just talk about "how to optimize a D program" b/c that's what this article is.

View on HN

In general, Python is slow (compared to C or whatever) because of excessive memory allocation and overuse of hash maps.

PyPy probably manages to optimize the hash map/method call lookups for these small programs, which explains the speedups. Removing memory allocations is still hard.

The D language provides finer mechanisms to control memory and data structures. This makes the language larger, but enables you to optimize if it becomes necessary.

Still, I agree and I would like to see a Python expert to optimize it.

> In general, Python is slow (compared to C or whatever) because of excessive memory allocation and overuse of hash maps.

Python is a highly dynamic language with an API (towards both Python and C) that is very invasive. These two things, taken together, make optimizing the interpreter extremely difficult, because practically all of it can be modified or introspected. CPython being implemented largely as a hashtable-interpreter is only one facet to its performance.

Perhaps a talk recommendation: https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seL...

The article wasn't really an attempt to show how D is "faster" or "better" than Python. I think the author was trying to baseline code size and relevance to the problem by illustrating how it compares to a typical Python solution.

I think this repository gives more context as to why they might of showed some Python code:

https://github.com/eBay/tsv-utils-dlang

They basically explored new languages to rewrite some perl script in and liked D enough to shift over. They have other tooling in other languages, my guess is they'll unify a good amount of it in D. Disclaimer: this is based on my own assumption that they like D so much they want to just use it all over. It wouldn't surprise me to see this confirmed by an eBay employee. Although seeing as he wrote TSV Utilities, it really wouldn't surprise me if he wants to rewrite all in-house tooling he uses in D as the repository states.

You're right... naive PyPy and D are in the same ballpark. Then they only optimized the D version.

I was excited with D's performance before I realized it is barely faster than PyPy. Almost not much of a point unless it saves in other ways like concurrency and parallelism?

This problem is not designed to compare language speed. You can get C speeds (and sometimes better) using D.

Not to mention using .split(delim) instead of the proper CSV parsing library that ships with Python.

Out of curiosity, I gave it a shot. I came out roughly 20% faster using python's inbuilt csv library.

When I switched to pypy the csv library actually made it nearly 2x slower than pypy using .split(delim)

And you were using the pure Python CSV library, not the C one?

How can you tell?

I used the stdlib one, just straight "import csv"

Of course it's faster. Sorry, I wasn't clear. But good luck handling a tab in a quoted field, then.

Reply on news.ycombinator.com