HNHacker News
TopNewBestAskShowJobs

EPendragon

195 karma · joined June 6, 2025

I enjoy developing software and exploring new tech.

Blog: https://evgeniipendragon.com/

submissionscomments
EPendragon··on I was wrong about robots.txt
I mean it was a realization for me, although I wouldn't call it profound. To your point, it was closer to obliviousness, which led me to learn more about Open Graph Protocol details and how Robots Exclusion Protocol works.

I try to write about things that I learn or find interesting. Sharing it here in the hopes that others might find it interesting too.

EPendragon··on I was wrong about robots.txt
I like the idea of using HEAD or OPTIONS methods and have all bots access that so that they get a high level idea of what's going on, without the access to actual content, if the owner decided to block it.
EPendragon··on I was wrong about robots.txt
I do like your suggestion of creating some standard that categorizes using function or purpose like you mention. This could simplify things granted that there is a way to validate the function and for spoofing to be hard to achieve. And yes - there is also legal.

I do think that I will likely need to go back and unblock a couple of other bots for this exact reason - so that it would be possible to share it and have previews in other social media. I like to take a slow and thoughtful approach to allowing this traffic as I get to learn what it is that I want and do not want.

Comments here have been a great resource to learn more about this issue and see what other people value.

EPendragon··on I was wrong about robots.txt
I think a better solution would be to block all the traffic, but have a comment in robots.txt with a way to be added onto a whitelist to scrape the contents of the resource. This puts a burden of requesting the access on the owner of the bot, and if they really want that access, they can communicate it and we can work it out.
EPendragon··on I was wrong about robots.txt
There are tools like curl-impersonate: https://github.com/lwthiker/curl-impersonate out there that allow you to pretend to be any browser you like. Might take a bit of trial and error, but this mechanism could be bypassed with some persistence in identifying what is it that the resource is trying to block.
EPendragon··on I was wrong about robots.txt
That was my hunch. My initial post on robots.txt: https://evgeniipendragon.com/posts/i-am-disallowing-all-craw... - revolved around blocking AI models from doing that because I do not believe that it will bring more traffic to my website - it will use the content to keep people using their service. I might be proven wrong in the future, but I do not see why they would want to let go of an extra opportunity to increase retention.
EPendragon··on I was wrong about robots.txt
For me (as I'm sure for plenty other people as well) limiting traffic to actual users matters a lot because I'm using a free tier for hosting in the time being. Bots could quickly exhaust it, and your website could be unavailable for the rest of the current "free billing" cycle, i.e. until your quota gets renewed.
EPendragon··on I was wrong about robots.txt
It is awesome to see that there are efforts to improve upon the old standards with modern web security standards
EPendragon··on I was wrong about robots.txt
I definitely agree here. My initial response was to block everything, however you realize that web is complex and interdependent. I still believe that everyone should have autonomy over their online resources if they desire. But that comes with an intentionality behind it. If you want to allow or disallow certain traffic, you also should answer the question why or why not. That requires understanding what each bot does. That takes time and effort.

My foray into robots.txt started from the whole notion of AI companies training on everything they can put their hands on. I want to be able to have a say whether I allow it or not. While not all bots will honor the robots.txt file, there are plenty that do. One way that I found you can test that is by asking the model directly to scrape a particular link (assuming the model has browsing capabilities).

Bots are not malicious by default. It is what that company does with your data and how you feel about it that matters in the end.

EPendragon··on I was wrong about robots.txt
Would robot traffic be considered tracking in light of GDPR standards? As far as I know there are no regulatory rules in relation to enforcing robots behaviors outside of robots.txt, which is more of an honor system.
EPendragon··on I was wrong about robots.txt
I kind of agree. The rules for safety should be simple, straightforward, and protect you in the "edge cases", i.e. following while not paying 100% of attention, protect you with a malicious actor in mind aka reckless driver, etc. Ideally, in a system like that it should be a difficult and intentional behavior if one wanted to break the rules rather than to follow them.
EPendragon··on I was wrong about robots.txt
I believe there is a difference between education and critical thinking. One may not have a certain level of education, but could exercise a great degree of critical thinking. I think that education can help you understand the context of the problem better. But there are also plenty of people who are not asking the right questions or not asking questions - period - who have lots of education behind them. Ironically, sometimes education is the path that leads to blind trust and lack of challenging the status quo.
EPendragon··on Show HN: MidWord – A Word-Guessing Game
I played this game for a bit. Really loved it! Later, I realized that it is possible to get the secret word using JS in the console: https://evgeniipendragon.com/posts/reverse-engineering-midwo...
← PreviousPage 3 of 3