Hacker News Reader

Comment by inertiatic | Hacker News Reader

Defining custom sort orders in Solr is as simple as uploading a text file with the values you intend to use for ranking. This is a great feature that is in fact missing from Elasticsearch and saves you so much reindexing time.

There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish.

View on HN

> There certainly are usecases where Lucene based solutions aren't the best fit. But I think the claim that you couldn't make something faster by moving away from Python is outlandish.

I read that as a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C. I am surprised that you read it that way. Who in their right mind would say Python speed is faster than C speed?

You read

>I doubt you could have made it faster even writing the entire thing directly in C or C++.

> a statement that they implemented a proper and bespoke algorithm, not that the speed of Python is greater than C.

You seem fundamentally confused, for example with tools like Cython.

Many extension module implementations in Python are literally as fast as pure C (not just nearly as fast with minor extra CPython overhead, but literally as fast as pure C by deliberately bypassing CPython VM loop and data models).

Cython can be as fast as C for simple use cases.

Because you're writing regular Python for a production service though, and not artificially writing optimized examples, then you will occasionally have to pay extra costs.

Are you perhaps a bit too invested in your own narrative?

Yes. If the implementation language isn't the determining factor for speed, then what is the cause? There is a branch of Computer Science called Algorithmics[0], wherein one expression of measurement is called Big-O notation[1].

[0] https://en.wikipedia.org/wiki/Algorithmics

[1] https://en.wikipedia.org/wiki/Big_O_notation

You are incorrect. This is only true if you can precompute all sort orders (many types of sort orders cannot be precomputed and depend on additional context data only available at query time, especially for personalization or trending solutions). Additionally you must deal with precommitment to very poor sharding properties with Solr. With our Python approach, we could hold all results (billions of content items with trimmed down data structures representing only the thinnest container needed for each sort order representation) in memory easily, and dynamically resort on the fly or double sort by multiple sort orders that each required contextual data only available at query time.

Reply on news.ycombinator.com