220 karma · joined December 10, 2017
Turns out the page was ending up in bfcache (https://web.dev/articles/bfcache), which in turn led to the button's loading state being preserved when clicking back. Listening to the `pageshow` event enables one to handle that somewhat gracefully.
> What was the biggest thing you learned while implementing this?
How much the quality of the data and resulting indexes matter. My impression based on this experience is that "RAG" might be a cohesive set of techniques, but their application to various domains likely is very domain-specific.
> Was anything surprisingly difficult?
Evaluating results is very tedious, almost by definition: you need to figure out ground truth by some mechanism and build evaluation datasets from there. To be honest, a lot of this beta was built on "vibes" only.
> Was there anything that worked better than you expected?
In terms of whether something worked better than I expected: modern embeddings are really magical. I'd previously worked with TF/IDF (a decade-or-so ago) and Doc2Vec (6-7 years ago), and while those were surprisingly useful, they really pale compared to what LLM embeddings can encode in very dense representations.
Quality aside, I think the primary challenge is in figuring out the right UX for delivering that at scale. One of the really great advantages of Google is that it is right there in your URL bar, and that for many of the searches you might do, it works just fine. Figuring out when it doesn't and how to provide better result then seems like a big unsolved UX component of figuring out personalized search.
Yup, totally feasible. I might add that!
To learn your interests we'd at a minimum need to know what HN stories you tend to click or comment on, e.g. by a different reader view or using a browser extension. Presumably your comments and submissions could provide useful signal as well :)
Keyword searches on the Algolia engine will generally result in better recall -- at least when identifying the right keyword is easy, e.g. the name of a company. They likely will require more sifting through results & keyword "engineering" however.
In my mind the two approaches are complementary. I suppose there's an argument for working more directly towards blending them :)
I'm planning to fix that in short order, feel free to sign up at https://hackersearch.net/signup if you care to receive an update when that goes live!
I definitely want to re-explore that though; I think it should be possible to do so a lot more rigorously now that I have a better sense for what people want to search for.
I actually generate two summaries: one is part of the ingestion pipeline and used for indexing and embedding, and another is generated on-the-fly based on user queries (the goal there is to "reconcile" the user query with each individual item being suggested).
I use GPT-3.5 Turbo, which works well enough for that purpose. Cost of generating the original summaries from raw page contents came down to about $0.01 per item. That could add up quickly, but I was lucky enough to have some OpenAI credits laying around so I didn't have to think much about this or explore alternative options.
GPT-4 would produce nicer summaries for the user-facing portion, but the latency and costs are too high for production. With GPT-3.5 however those are super cheap since they require very few tokens (they operate off of the original summaries mentioned above).
Worth noting that I've processed stories by score descending, and didn't process anything under 50 points which substantially reduced the number of tokens to process.
In the meantime, one thing that comes to mind is that simply embedding the whole contents of the webpages after scraping them didn't yield very good search results. As an example, an article about Python might only mention Python by name once. I found that trimming extraneous strings (e.g. menus, share links), and then extracting key themes + embedding those directly yielded much, much better results.
One big distinction with the "site:https://news.ycombinator.com" hack is that the search on Hacker Search directly runs against the underlying link's contents, rather than whatever happens to be on HN. We also more directly leverage HN's curation by factoring in scores.
Appreciate your suggestions; will look into building those!
If you feel up for it, you should share your email in the righthand "Unhappy with your results?" widget. My plan is to manually look into the disappointing searches and follow-up with better results for folks, in addition to fixing whatever can be fixed.
Agreed re: searching comments (which it indeed currently doesn't do).