If smaller indie hackers are going to build useful competitors, they’re going to need to scrape.
I understand that a lot of scraping is from spammers, but not all.
If smaller indie hackers are going to build useful competitors, they’re going to need to scrape.
I understand that a lot of scraping is from spammers, but not all.
Any useful competitor will have to scale well beyond the point of indie hacker, where someone is (eventually) getting rich. That doesn't make for a great argument for bypassing consent.
Edit: As mmcclure points out Kagi actually doesn't index much. So that's a bad example that might invalidate my point.
> Our data includes anonymized API calls to traditional search indexes like Google, Yandex, Mojeek and Brave, specialized search engines like Marginalia, and sources of vertical information like Apple, Wikipedia, Open Meteo, Yelp, TripAdvisor and other APIs. Typically every search query on Kagi will call a number of different sources at the same time, all with the purpose of bringing the best possible search results to the user.
source: https://help.kagi.com/kagi/search-details/search-sources.htm...
both OpenAI and Google rely on scraping as their foundation for…everything.
OpenAI and Google also (claim to) respect the most basic bot deterrent, robots.txt, with their crawlers. If someone doesn't want their content scraped to be included by search indexes or LLMs/models, then that certainly includes "smaller indie hackers."