While I agree that for user-facing APIs, serverless can both help handling load spikes and keep costs low, how would serverless help with spidering and indexing?

Spidering and indexing are processes that would most likely run continuously for any search engine like Google or Kagi. There is always data to update and new web pages being created.

Thus, they would benefit from dedicated servers from the get go for costs. Ahrefs posted an interesting article on this topic: https://tech.ahrefs.com/how-ahrefs-saved-us-400m-in-3-years-...

Then again, you could use serverless on your own dedicated servers to benefit from the advantages of both.