How? What's the incentive structure for that design? There are no extra ads to be displayed here, so how does Meta get value from that design? Seems way more likely to be incompetence in my world view. Explain why that design is superior!
Someone puts a high number of pages crawled per minute in their PSC. Doesn't matter if they're 403s.
There are no extra ads to be had even by scanning sites with a robots.txt that allows them, because the value of pages could never have been in displaying ads but must have resided in the information on the pages itself.
With Muse, people can enter their bank account information and medical information so that it can work hard for them. For this, they need a lot of training material.
LLM training data
Reverse Hanlon's Razor
Their apps and their hardware might be gorefests but you best believe their tracking software is rock solid with no possible failure modes