Hacker News Reader

It's not exactly secret:

https://support.apple.com/en-jo/HT204683

"Applebot is the web crawler for Apple. Products like Siri and Spotlight Suggestions use Applebot."

It's crawling things for me at work that I cannot imagine being used for Siri or Spotlight Suggestions, though

Isn't the point of a crawler to crawl nearly everything? Of course they might weigh some things much heigher (recrawl wikipedia much more frequently, for example), but you don't know what might be interesting as a search result until you've crawled it!

How is it possibly going to know whether something is appropriate for those unless it crawls them first?

Reply on news.ycombinator.com