HNHacker News
TopNewBestAskShowJobs

costco

2,675 karma · joined December 30, 2019

chrisjtarry \at\ gmail.com

https://github.com/chris124567

submissionscomments
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
I did not add what you requested exactly because I think in many cases authors have written less popular books that people may not be aware of but if you try again you should see less highly repetitive things like 5 of the same series in a row in the results.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
I just introduced something [1] which should give you much less repetitive recommendations if you want to try again.

[1] https://www.cs.cmu.edu/~jgc/publication/The_Use_MMR_Diversit...

costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
Hi, I just added something called maximal marginal relevance which should give you much less repetitive recommendations.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
Do you just adding the similar button that the input books have to the output books?
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
What do you think the probability that someone else read 15 books you also read is? It’s very unlikely unless they are all staples of a genre, part of the same series, or just extremely popular in general. 3-5 books is how much I would use on that page. I have found interesting accounts of medievalists, people who work at think tanks, etc with it.

Fake users I would agree should be filtered, but I don’t think filtering out users who gave it a bad review is necessarily the intended behavior. If I put in 3 semi obscure Russian history books, I am presumably looking for someone who is an expert in Russian history to see what else they read. In that case I don’t care if they didn’t like one of the books or not. Approximate matches would require something like LSH or cosine similarity of average input book embedding against average embedding of read books of every user which I think wouldn’t work well anyone for retrieving anyone with a moderately long interaction history.

costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
Yeah, the latter are just because they are popular. If you have 3+ books you tend to get less random popular books included
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
You may find these better:

https://book.sv/similar?id=211570

> Note 1: If you only provide one or two books, the model doesn't have a lot to work with and may include a handful of somewhat unrelated popular books in the results. If you want recommendations based on just one book, click the "Similar" button next to the book after adding it to the input book list on the recommendations page.

costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
This is a result of the use of positional embeddings, which typically results in the final item being weighted very highly. The problem is that this information is shown to be very relevant to the task of predicting the next item interacted with. If you add more books the effect of this is somewhat diluted.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
Thank you for the kind remarks. I would say your request is reasonable and it’s something I thought about but it’s worth noting that if you scroll down on a Goodreads book page you can see all of the users who gave it 1 star, 2 stars, etc (not just reviews, ratings too). And in fact Goodreads does not tell people this but these lists include private users. So someone looking to cause controversy would probably just find an incendiary book and look for everyone who gave it 5 stars. My data does not include people who marked their account as private or only visible to those who are signed in and also does not give information on stars. It is also very easy to remove your data if you are so inclined. I only log request URLs but I can tell that many thousands of people visited the site and less than 10 opted out despite “Remove My Data” being prominently displayed and referenced in the attached text to my post.

Also, I don’t use them but based on forum posts I read I think other services like LibraryThing or Storygraph do expose similar information about book readers.

costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
You don't have to import your Goodreads profile. You can type titles and authors in the box and find books to add to the list that way.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
I would agree the results are generally OK but do not feel magical in most cases (I think in some specific cases they do though). The results can be not great if you add books across many disciplines. For instance if you add "The Elements of Typographic Style" and "The Design of Everyday Things" (https://book.sv/#671857,18518), you do get "Grid Systems in Graphic Design" but under its German name "Rastersysteme für die visuelle Gestaltung."
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
> Note 1: If you only provide one or two books, the model doesn't have a lot to work with and may include a handful of somewhat unrelated popular books in the results. If you want recommendations based on just one book, click the "Similar" button next to the book after adding it to the input book list on the recommendations page.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
I use `fetch` on relative endpoints so that's odd. There shouldn't be any external API calls on my website other than whatever the Cloudflare captcha uses. I also use HTTPS-only in Chrome and did not experience any issues. I just tested Firefox with HTTPS-only on/off and Safari on my phone and I was able to import shelves for multiple users. Are you sure that you do not have any privacy settings on (can you access your shelf in Incognito mode)?
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
I use a Hetzner server with Ryzen 7 3700X and an SSD.

I think I could get the model to work with ONNX web but it'd be a 2GB download so the user experience wouldn't be too great. My Meilisearch index is ~40GB but I don't know how much that could be compressed down.

Here's how the similar page for books is generated, which I forgot to mention on the "how it works" page: https://gist.github.com/chris124567/8d06d64bfe827cb7f6121f93...

costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
It's recommended that you put at least 3 books in. If you would like recommendations just based on one book, click the similar button on the book, it should take you to this page: https://book.sv/similar?id=297914
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
You can import the first or last 64 books of your read, to-read, or currently-reading shelves if you press the "Import Goodreads" button and provide your Goodreads ID.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
Thank you for the recommendations. I didn't try BERT4Rec because I assumed it would perform the same or worse as what I already had after having read https://dl.acm.org/doi/pdf/10.1145/3699521. The TIGER paper seems interesting - I definitely want to explore semantic IDs in general and also because I think it could allow including more long-tail items.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
Worked for me, could be due to server being overwhelmed

Here is the URL with your books: https://book.sv/#52752877,46049530,18437030,52480873,3260654...

costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
Yes I would say the handling of series is probably the biggest problem. Once my test metrics got to a point I was happy with and my quality spot checks passed (can I follow the models recommendations from one generic history book to Steven Runciman, also making sure popular books don't always dominate the results), I was ready to release because I had been working on this project for so long. The solution is probably using the transformer model to generate 100-200 candidates and then having a reranker on top.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
I didn't want an agent to get stuck on an infinite loop invoking endpoints that cost GPU resources. Those fears are probably unfounded, so if people really cared I could remove those. /similar is blocked by default because I don't want 500000 "similar books for" pages to pollute the search results for my website but I do not mind if people scrape those pages.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
If you want recommendations solely based on one book, please try the similar page: https://book.sv/similar?id=13566692

These seem to fit the description you are going for better. The model is trained to predict the next book in the sequence. Those other books you listed happen to be very popular, so in the absence of information about you (only having 1 book), the model will tend to recommend those.

costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
Everything (namely Meilisearch, Postgres and the web server in Go) besides the model inference is running on a Hetzner server with a large SSD and an "AMD Ryzen 7 3700X 8-Core Processor." The data.ms directory is about 40GB. Once the HN traffic dies down I will probably move the model back to the Hetzner server so I don't have to pay $0.15/hour for an A4000.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
Not sure if I can. At the very least book descriptions most likely could not be distributed. There is an academic dataset with around 200M reviews though: https://cseweb.ucsd.edu/~jmcauley/datasets/goodreads.html
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
I think I will expand the input books limit (sadly requires retraining) and or the output books limit of 30.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
I'm not an expert by any means but as far as sequential recommendations go, aren't SASRec and its derivatives pretty much the name of the game? I probably should have looked into HSTUs more. Also this / sparse transformers in general: https://arxiv.org/pdf/2212.04120
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
It's explicitly trained to predict the next book read in a sequence, which is why you get that behavior. There's probably a better way for me to handle it rather than having 5 books from the same series tend towards the top though.
costco··on Show HN: I scraped 3B Goodreads reviews to train a better recommendation model
Thank you for the compliments :) I used 50-100 datacenter proxies. I just logged requests made by the iOS app with Charles and then recreated the headers to the best of my ability though the server did not seem to be very strict at all. Worth noting though that static residential proxies are not too expensive these days anyways.

Re the API: The model does actually run fairly well on CPU so it probably wouldn't be too expensive to serve. I guess if there is demand for it I could do it. I think most social book sites would probably like to own their recommendation system though.

costco··on 'SIM Farms' Are a Spam Plague. A Giant One in NY Threatened US Infrastructure
Finally someone in this thread who gets it. In fact many of the temporary phone number registration services used to make bot accounts have pages on their website where they say "Do you live in country X and have access to large amounts of SIM cards? Partner with us - we will send you the equipment." Calling the US through standard means is so cheap that I suspect the only reason anyone would use a grey market provider is if they were making spam calls, but maybe I'm wrong.
costco··on NYC Telecom Raid: What's Up with Those Weird SIM Banks?
I've read that at least for for GSM VoIP gateway setups, people typically rotate SIMs because it looks suspicious for one customer to be making calls all day. In fact there is a whole industry most people have no idea about dedicated to detecting these so-called grey routes. Having perused some of these offerings, it seems fairly typical to offer 16-32 radios and 256 SIM card slots. For residential proxy setups, I've seen a bunch of 16 port USB hubs with Huawei LTE modems.
costco··on How did .agakhan, .ismaili and .imamat get their own TLDs?
Related: It’s only about $5k a year to run your own registrar (which is different than a gTLD). You have to sign agreements with Verisign and others to register domains on TLDs that people actually want but it’s not unduly hard and I think they even offer good payment terms.
← PreviousPage 2 of 13Next →