I agree. Collaborative filtering should succeed when you have sufficient of data, but we don't always have sufficient data. I think books are a tough challenge: the number of books is relatively large (vs. e.g. the number of restaurants on Yelp), the number of books people read is relatively small (vs. e.g. how many things they buy). Plus, people's tastes in them are relatively particular, and they're a big time commitment, so the bar for what counts as a "good" recommendation is high.

I think in these cases you can do better by bringing in some content-based filtering. I made an experimental book recommender using only story trope tags and I thought the results were already better than what I was getting elsewhere. It's still up at https://bookslikethis.herokuapp.com/ (but it basically only has sci-fi/fantasy titles).