HNHacker News
TopNewBestAskShowJobs

huahaiy

351 karma · joined November 7, 2011

I am a technologist, with an academic background in psychology and computer science.
submissionscomments
huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
BTW, you were with me? Where?

Why do you have to use a throwaway account to say this, I am curious.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
For what its worth, a final advise for those fresh graduates and college students. Don't take too seriously what your learned from books in school. They are not the gold standard the society measures things by. You got the wrong impression if you think your professors taught you that. They knew better. Don't take too much pride in your degree or in your specializations. They are nothing.

If you do not learn this now, your managers who maybe only has an associate degree from a community college, or from a school you look down on, will make you learn that soon enough.

These are not important things. An important thing is to keep an open mind and keep learning, so you are continuously employable. You can congratulate yourself when you are as old as me and can still do new things.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Thank you. I am glad that you learned something from the article. I wrote the article for people like you, who are seeking knowledge and self improvement.
huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Did I claim that? Straw man much?
huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Not my nickname, my name.

It's good for you that you are not a racist (it's sad that such a normal thing has to be a compliment, but it is a compliment. Good for you.), but racists do look for these cues and make their judgement based on these.

It's an unfortunate current state of affairs that we have to live with. Ignoring it does not make it going away though. If you don't like it, you got to call it out whenever you see it, which I hope you do, instead of being taken back by it, as you seems to be.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
That's the problem right there. The so called "Information Retrieval Community" does not seem to care about what users want, and they treat their users in a condescending way, ignoring an obvious problem for years, as clearly demonstrated here and elsewhere.

Why should I care about what they think? In the end of the day, they are serving the users. They are either going to be left behind, or they have to change.

For example, the newer search engines, such us, https://github.com/meilisearch/MeiliSearch, also do what T-Wand does. Users are going to vote by their feet. The "information retrieval community" be damned. It's as simple as that. Got it?

And change they will. What "information retrieval community" are you talking about? Just a bunch of fresh graduates who just got their PH.D, and some college students. That's about it. I don't see any IR professors coming out defending them.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
As I have repeated suggested, since you obviously know Lucene, it does not take much for you to write a few lines of Java code to say, "here, this is how it is supposed to be done".

Instead, you keeps giving all kind of excuses. You said you do not have time, but somehow you have time to write long wall of text like this. If you are not trolling, what you are doing then?

I pointed out to you why your suggestions will not make differences, some I have already tried. For example, initializing a query parser only once, but that would crash Lucene. Clearly, you do not know about this, so your knowledge of Lucene is not as good as you think you know.

Fair?

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Let me offer my arguably biased insight: racism.

I am obviously a Chinese.

In the eyes of racists, Chinese Americans are supposed to be timid and does not make any noises, but I do make noises, so I am perceived as a smug today because I said I am a Ph.D, something else next time I said something else.

It's not complicated.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
You are making up straw man again. Did I make that claim?

I was simply motivate my work, pointing out it's a problem that the current generation of search engine does not address.

What is "relevance", it is of course a context sensitive question.

You talk as if BM25 is the gold standard when it is not.

The research on this is all over the place. I just read an article that says that BM25 is way worse than alternative language models. You don't have to look far, for example this one that talks about Wand:

https://dl.acm.org/doi/10.1145/2537734.2537744

You know why I quit academia? It is useless arguments and virtual signalings like these.

I'd rather go out and build a damn thing that people like to use.

I merely pointed out that there's a problem, and I have a solution. I did not claim that my problem and my solution solve all problems. Isn't this obvious?

In my problem, my solution beats Lucene. It's as simple as that.

So if Lucene wants to be this infinitely configurable search library, it would be advisable to offer my solution as an option, or offer an better one that does something similar.

So far I have not seen any takers, only excuses.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
You either has or has no guarantee, that's what the word "guarantee" means. Don't mince words.

You had a wrong understanding. Now you are corrected. Let's move on.

Nobody said anything about "fewer term matches could not be more relevant". You are just making up straw men here. That's not the discussion we are having.

What I said is this, from a user point of view, it's not good to have a document containing fewer query terms to rank higher. This is a fact that even Lucene acknowledge (at least when they were version 3.5.0.). You have nothing to counter this fact.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
I think I am pretty qualified to make my declaration, since it is about user experience.

I also do know a lot about IR, because I taught Information Retrieval class for 3 years when I was teaching in university, I read research papers, and I just come up with a new search algorithm.

It's just that some people cannot accept that there are people who can cross fields with ease, make contributions quickly, and move on to the next field that pit their interests.

Yes, I am one of those people. In addition to HCI, I also published in the following areas: VR, DB, NLP, IR and Psychology. Sorry to hurt your feelings, but it is what it is. Accept it and move on.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
There's no guarantee in pure vector space model that it is the case. Your understanding is way off. This is a caveat that Lucene prominently put on their Web page.

https://lucene.apache.org/core/3_5_0/scoring.html

"Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on java-user@lucene.apache.org to figure out why a document with five of our query terms scores lower than a different document with only one of the query terms."

See? Lucene people know about it but they just do not think it is a problem.

But I do.

T-Wand, though also uses vector space model, makes sure it is the case.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
I do not want to tweak Lucene. I don't want a huge dependency. I don't want Lucene. Period.

I am writing a database, full-text search is just a *minor* feature.

Why should I bringing in a huge dependency for a minor feature when my whole code base is orders magnitude smaller than that?

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Feeling smug?

You fundamentally misunderstand both Wand and T-Wand.

I suggest you to read my article and the papers again.

The novelty of T-Wand, is not in sorting the rows on idf, which T-Wand actually does NOT do, but in adding some filtering conditions that are not previously reported in the literature.

You missed that, so you missed the whole point. Smugness hurts.

I see that you work on Facebook Search and I know you are one of the authors in this field, I can see why your feelings are hurt.

It's okay. There are always people smarter than you are, who can come into a completely new field, spend a few weeks, and come up with something new that you could not come up with when you have spent your whole career on.

It's OK. When you are older, you will learn to accept that. You can only work with what life gives you.

Please take my ideas and put that in Faceook Search, because it presently sucks.

BTW, you should have put the PDF of your paper on the Web, then I would have read it, instead of only recognizing your name, put it on arxiv like everyone else, if you want people to read it.

Also, that's why I post it on hacker news instead of publishing first in a conference. Because there will be more people reading it. Not many people read papers.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
I don't think he misunderstood anything.

There's the kind of library Lucene wants to be, a configurable library. But there are other libraries do not want that.

In this case, Datalevin is a database system, full-text search is just but a feature, I do not want users to configure full-text search in Datalevin as much as they like in Lucene. I want the default to be good, which Lucene's is not.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
It's your loss then.

It is actually smug, to suggest the author, in this case, me, a computer scientist and a past professor who taught Information Retrieval class for more than 3 years, and who just came up with a new search algorithm, to "take a look at Wikipedia page for TF-IDF".

In case you have not read the article due to your smugness, my search algorithm also uses TF-IDF and vector space model.

Please do not dis-understand what is going on here: I am running a startup, and I am also old enough to not care about publications as much as people who are younger or in academia.

That's why I chose to reveal this in a blog post instead of hiding it until after my paper is published. Understood?

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
It would really be a sad joke if one dismisses a good technology just because he does not like the tone of the author of the said technology.

Besides, the said technology does do tf-idf and fuzzy search, and will soon have phrase search, and other features, if enough people request them.

Datalevin is an open source project, not unlike Lucene. What's the difference? Not like the tone?

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Incorrect. The benchmark is fine. Your so-called "fundamental" flaws are just superficial things, such as using a different time measure, using a different benchmark library, passing in a thread pool, and so on. These are immaterial for relative comparison, unless proven otherwise.

I have repeated asked you to send the code that does things properly in your mind, and you refused to do so.

So you are just trolling. Please stop hinder the progress of the industry through pointless trolling. Makes some positive contributions instead, and I have pointed out repeatedly how to make a positive contribution in this case. Please do.

You are a university student, I am a seasoned computer scientist, past researcher and professor, I am busying running a startup. This will be the last time I will say this to you: please consider contributing something positive to the world.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
It does have relevance to user experience, which is under discussion.
huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
BM25 is just a variant of tf-idf.
huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Corrected. Thanks. I am not a native English speaker, so please do not read too much into it.
huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Show me how to do that? I will integrate that into the benchmark. Thanks.
huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Merged your PR. Thank you! If you do not know Clojure, that's fine, just outline what you want to do in Java, I can translate that into Clojure. Thanks.

To be honest, all your proposed changes would not make material difference, for algorithmic differences cannot be made up by implementation details.

As has already been demonstrated, T-Wand is implemented in a much slower language, Clojure. Clojure is implemented on top of JVM, it cannot possibly beat the same algorithm written in optimized Java. The only way to make a difference is to change the algorithm. So all your suggestions would not make a difference.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
I would appreciate if you also send a PR to address these issues, since clearly you know more about benchmarking than me.

Contributing to open source project is good, right?

BTW, Lucene is an open source project, Datalevin is also an open source project. Contributing to either would be equally good.

Even if you only want to contribute to Lucene for some reasons, a better contribution would be to integrate T-Wand in Lucene, rather than trying to talk down Datalevin. Would you agree?

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Send a PR with Lucene properly used. It's an open source project, and I said I welcome any contribution in the article. Please do.
huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Correct. That is also what all search engines will currently do.
huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
Send a PR if you feel the benchmark can be improved. I will be happy to merge it and rerun the benchmark.
huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
The industry has not "chosen" to not do what T-Wand does, the industry did not have that option because T-Wand just comes out. Now with T-Wand, the industry has that choice to make.

It is easy to implement T-Wand in the existing Lucene code base, and I would hope so. Otherwise, it would be a shame.

huahaiy··on T-Wand: beating Lucene in less than 600 lines of code
I would appreciate people who claims that "the author does not know Lucene enough" to send a Pull Request, where Lucene is configured to do the same thing that T-Wand does, that is to guarantee documents containing more query terms to rank higher than those containing less query terms.

To those who says "I like Lucene because it does Tf-idf", please be assured that T-Wand also does tf-idf and uses vector space model.

To those who say "T-Wand is just Wand", please be reminded that you are just repeating what the article was saying. Of course T-Wand is Wand, but you miss the technical contribution of T-Wand, which is clearly laid out in "What's new in T-Wand" section of the article. Of course, it is understandable if you are not able to judge the merit of that contribution, for that's the job of the peer review panel, which I do not expect Hacker News readers to be. This work will be submitted to a peer reviewed conference with more experiments added and all trade-offs discussed.

This may come as a disappointment to some, but the paper will not have a comparison with Lucene, for it would be scientifically unsound.

However, it should be easy for Lucene to add T-Wand as an option, and I do hope so.

huahaiy··on How much can a Clojure developer do alone?
Then it is actually far worse than what I guessed. You know next to nothing about Clojure, but somehow feel that you are qualified to have an opinion about it. That doesn't look good, does it?
← PreviousPage 3 of 7Next →