Hacker News Reader

Comment by nhggfu | Hacker News Reader

does your bot respect robots.txt directives?

pray tell, what is your bot's user agent string, [so i can nicely block you from my web-properties.]

I tried with a nonexistent path, here are the user agents I got:

  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

It's apparently built to evade detection.

the more reasonable assumption is that it uses a third party crawl service.

but still, it doesn't matter, it's acting on behalf of a user, and you aren't entitled to know what software your users run. One consequence of putting stuff on the public internet is that it's like public.

what sort of evasion technique is being used here, random user agents on each attempt?

> what sort of evasion technique is being used here

Using a user-agent that looks like a desktop user-agent, rather than including the name of the actual project/product that is being used.

I understand why though, plenty of websites block anything that looks like non-desktop/mobile user-agents, so makes sense. Besides the pragmatic reason, I also agree with menacingly that it doesn't matter, people should be able to use whatever user-agent they want to access the content you've put publicly on the internet.

The goal of SiteProfile is not to scrape data. It only accesses publicly available web pages, such as the homepage, about page, and pricing page. It does not access non-public content on websites, nor does it offer users the functionality to scrape website data.

What does it mean exactly for the service to provide information about a website without scraping it? How could summaries or LLM responses be generated be made without scraping pages?

Presumably the same way that Firefox makes an HTTP request to the webserver then formats the page for the human user. This is just formatting that page differently. This is no more a scraper than is Firefox's Reader Mode.

That said, lying about the UA is not cool.

I have something that sends a UA of "Sitetruth.com site rating system". Many sites won't talk to that.

I've used a reader mode library that I think as created by Mozilla and handles converting a site to reader mode locally. Does the Firefox browser do it locally, or at least on demand? If so I wouldn't really consider that scraping since they aren't parsing the site and storing data for later use.

It does scrape the site in order to summarise it, no?

Your statement doesn't answer their question.

I meant that it is not supported yet. I will add this to the to-do list, and I believe it does not conflict with the goals of SiteProfile. Thank you for your feedback.

so you scrape + store + process contact info etc, presumably.

sounds like a privacy nightmare

no doubt this is not GDPR compliant.

no doubt this is not legal in some parts of the world - unless people can opt out and get their data removed.

> no doubt this is not GDPR compliant.

Unless the project is open source, no doubt you cannot know this. If they don't store any of those details anywhere (including not in logs) but just pass it along, GDPR won't apply.

> does your bot respect robots.txt directives?

Would be a bit strange if it did, as the service is not a crawler/robot by any measures.

Bit like asking if cURL is "respecting" robots.txt.

It's just another user-agent after all.

It is a service that seems to crawl a website for content and feed that content into some LLMs. It should absolutely respect robots.txt. This is exactly what robots.txt is used for, to tell automated crawlers of a website what they should and should not do.

I disagree - this is not a crawler that just blindly stumbles around any random website that it finds. It is more akin to a user agent. The only requests it makes are derived from specific instructions by the user to do so.

Having said that, people may use it as a crawler, just like you might be able to script Firefox to be a crawler, but it is not in itself a crawler.

It doesn't need to be blind stumbling around the web. But you might be right about only grabbing one page, and if you are then I agree that abiding by robots.txt is only going to upset a tiny minority. When they talk about websites it makes me think they are crawling to see all the pages linked to the homepage, because the asking questions part is extremely limited if all it does is look at one page. If they crawl, then I think they need to abide. If they don't, I think it's ok.

> It is a service that seems to crawl a website for content and feed that content into some LLMs.

It doesn't seem to work like that at all, to me.

As far as I understand, you give it a specific URL, and it extracts content from that URL and that URL only. A "crawl" would mean it would also follow links automatically, which I don't see any evidence of being done, from the landing page at least.

Reply on news.ycombinator.com