Ask HN: Why is there no adopted standard for web scraping?

1 pointdennisy4 comments

I feel many sites benefit from scraping, or are happy to allow it within certain bounds. My question is why have no standards emerged around how data is structured and accessed for web scraping?

I am aware of standards such as Open Graph - is this the closest we have gotten to a standard of machine accessible data on the web?

View on HN

Schema.org is probably the closest thing.

But, if people want to make their data available, an API is probably the best way.

Agree, but the API should have a standard - hence my question. Making your site available via an API would be awesome!

I few like there has been a lot of attempts at this; the semantic web, html5 semantic tags (article, header, ...), stuff like robots.txt. Developers really like to generalise.

Getting everyone to cooperate is the hard part.

Would you say the issue maybe is that there has not been an easy way to implement the various standards?

If we had this automated for web devs the uptake would be much better?

Reply on news.ycombinator.com