HNHacker News
TopNewBestAskShowJobs

jnnnthnn

220 karma · joined December 10, 2017

https://jnnnthnn.com
submissionscomments
jnnnthnn··on Show HN: I built a LLM-powered Ask HN: like Perplexity, but for HN comments
Yay! Thank you for trying it out, so glad it's been useful!
jnnnthnn··on Show HN: I built a LLM-powered Ask HN: like Perplexity, but for HN comments
Thank you for trying it out!
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Thanks! I couldn't see it on Chrome due to React Devtools interference.

Turns out the page was ending up in bfcache (https://web.dev/articles/bfcache), which in turn led to the button's loading state being preserved when clicking back. Listening to the `pageshow` event enables one to handle that somewhat gracefully.

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Blog post is now live: https://jnnnthnn.com/building-hacker-search.html
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Snap! I can't repro this. Which browser are you using?
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
There's a place for both, yes! I think Hacker Search should offer each (https://news.ycombinator.com/item?id=40241332) and ideally intelligently figuring out what to do based on your query.
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Thank you!
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Thanks for trying it out!

> What was the biggest thing you learned while implementing this?

How much the quality of the data and resulting indexes matter. My impression based on this experience is that "RAG" might be a cohesive set of techniques, but their application to various domains likely is very domain-specific.

> Was anything surprisingly difficult?

Evaluating results is very tedious, almost by definition: you need to figure out ground truth by some mechanism and build evaluation datasets from there. To be honest, a lot of this beta was built on "vibes" only.

> Was there anything that worked better than you expected?

In terms of whether something worked better than I expected: modern embeddings are really magical. I'd previously worked with TF/IDF (a decade-or-so ago) and Doc2Vec (6-7 years ago), and while those were surprisingly useful, they really pale compared to what LLM embeddings can encode in very dense representations.

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
I definitely think there are opportunities to provide more useful & personalized search than what Google offers for at least some queries.

Quality aside, I think the primary challenge is in figuring out the right UX for delivering that at scale. One of the really great advantages of Google is that it is right there in your URL bar, and that for many of the searches you might do, it works just fine. Figuring out when it doesn't and how to provide better result then seems like a big unsolved UX component of figuring out personalized search.

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Thanks!

Yup, totally feasible. I might add that!

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Thank you!

To learn your interests we'd at a minimum need to know what HN stories you tend to click or comment on, e.g. by a different reader view or using a browser extension. Presumably your comments and submissions could provide useful signal as well :)

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Only indexing stories for now, not yet comments. That too will get fixed!
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Try any of the examples listed on the landing page. You can easily access the HN/Algolia search equivalent by clicking on "Try Hacker News search instead" to the right (on desktop).

Keyword searches on the Algolia engine will generally result in better recall -- at least when identifying the right keyword is easy, e.g. the name of a company. They likely will require more sifting through results & keyword "engineering" however.

In my mind the two approaches are complementary. I suppose there's an argument for working more directly towards blending them :)

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Thanks for the feedback! I made a cutoff at stories with <50 points when ingesting content, and it looks like that cut off what you were looking for. Definitely not great.

I'm planning to fix that in short order, feel free to sign up at https://hackersearch.net/signup if you care to receive an update when that goes live!

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Totally! I've come to deeply dislike thumbs up/down UXes, so I'm collecting that by recording clicks on results!
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Relatedly, the "Follow up queries" in the righthand floating insert are an attempt at finding a decent balance between recovering from failures & giving users enough control on the queries themselves :) See https://hackersearch.net/search?q=postgres+clustering&period...
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Sweet! Thanks for sharing. A prior implementation had HyDE running on user searches, but I found the results to be hit-or-miss depending on the query type.

I definitely want to re-explore that though; I think it should be possible to do so a lot more rigorously now that I have a better sense for what people want to search for.

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Hi Simon! Big fan of your blog.

I actually generate two summaries: one is part of the ingestion pipeline and used for indexing and embedding, and another is generated on-the-fly based on user queries (the goal there is to "reconcile" the user query with each individual item being suggested).

I use GPT-3.5 Turbo, which works well enough for that purpose. Cost of generating the original summaries from raw page contents came down to about $0.01 per item. That could add up quickly, but I was lucky enough to have some OpenAI credits laying around so I didn't have to think much about this or explore alternative options.

GPT-4 would produce nicer summaries for the user-facing portion, but the latency and costs are too high for production. With GPT-3.5 however those are super cheap since they require very few tokens (they operate off of the original summaries mentioned above).

Worth noting that I've processed stories by score descending, and didn't process anything under 50 points which substantially reduced the number of tokens to process.

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
In fairness, it's probably impossible to unambiguously determine what the intended/desired interpretation is (though intuitively it seems like k-means should be lower likelihood)!
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Welp, I'm so glad I wrote "at least one person", and not "one person" in my note in the OP :D
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Fun! Thanks for sharing! It's super fast and the bias toward smaller websites really does surface interesting things. Very reminiscent of the Kagi small web site, also!
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
I suspect this is probably because of a bit of a bias in the indexed dataset: at present, the indexed stories tend to bias toward high-scores ones, and at a glance I don't see that many stories about Postgres clustering in that distribution.
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Ack! If you're willing to share that example with me over email (jonathan@unikowski.net), I'd love to see what we can do. Maybe good enough semantic search over comments would remove the need for filtering on post type?
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Thank you, and likewise!
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Sweet! I'll try running it tonight.
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Nice!
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Thanks! I tried a few other approaches and found the LLM results were overall better (latency and cost aside). Maybe that should be an option made available to users though...
jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
I'll write up a little blog post once the traffic dies down a bit!

In the meantime, one thing that comes to mind is that simply embedding the whole contents of the webpages after scraping them didn't yield very good search results. As an example, an article about Python might only mention Python by name once. I found that trimming extraneous strings (e.g. menus, share links), and then extracting key themes + embedding those directly yielded much, much better results.

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
Thanks for the feedback!

One big distinction with the "site:https://news.ycombinator.com" hack is that the search on Hacker Search directly runs against the underlying link's contents, rather than whatever happens to be on HN. We also more directly leverage HN's curation by factoring in scores.

Appreciate your suggestions; will look into building those!

jnnnthnn··on Show HN: Hacker Search – A semantic search engine for Hacker News
It is mostly RAG, although I suppose that doesn't say much about the system: one thing I've found is that the way you clean and process the data substantially changes the quality of the results. I'll write a little blog post sharing some of the learnings!

If you feel up for it, you should share your email in the righthand "Unhappy with your results?" widget. My plan is to manually look into the disappointing searches and follow-up with better results for folks, in addition to fixing whatever can be fixed.

Agreed re: searching comments (which it indeed currently doesn't do).

← PreviousPage 2 of 3Next →