I try to write about things that I learn or find interesting. Sharing it here in the hopes that others might find it interesting too.
195 karma · joined June 6, 2025
Blog: https://evgeniipendragon.com/
I try to write about things that I learn or find interesting. Sharing it here in the hopes that others might find it interesting too.
I do think that I will likely need to go back and unblock a couple of other bots for this exact reason - so that it would be possible to share it and have previews in other social media. I like to take a slow and thoughtful approach to allowing this traffic as I get to learn what it is that I want and do not want.
Comments here have been a great resource to learn more about this issue and see what other people value.
My foray into robots.txt started from the whole notion of AI companies training on everything they can put their hands on. I want to be able to have a say whether I allow it or not. While not all bots will honor the robots.txt file, there are plenty that do. One way that I found you can test that is by asking the model directly to scrape a particular link (assuming the model has browsing capabilities).
Bots are not malicious by default. It is what that company does with your data and how you feel about it that matters in the end.