HNHacker News
TopNewBestAskShowJobs

EPendragon

195 karma · joined June 6, 2025

I enjoy developing software and exploring new tech.

Blog: https://evgeniipendragon.com/

submissionscomments
EPendragon··on Customizing tmux
I will definitely check it out. Thanks for the suggestion! What would you say is different between tmux and Zellij, and why did you start using the latter?
EPendragon··on Customizing tmux
I have checked it out initially. It hd a good set of defaults. I didn’t like everything that came with it, but I did read through its config file and borrowed a couple of lines from there into my own setup. IMO the problem with using a prebuilt config like that is that it requires you to learn all of its keybinds, styling, layout. I would rather build my own incrementally but have a sense of knowing everything that is going on in my setup.
EPendragon··on Ask HN: What is the average karma points of HN users?
Im curious to know what is the karma to activity level ratio for the active members of the community? Are there more people who constantly post and comment or are there more people who create rare, but high value posts that the community values and engages with?
EPendragon··on Ask HN: What is the average karma points of HN users?
What’s your activity like? Do you post/comment on the daily basis?
EPendragon··on I was wrong about robots.txt
I started having similar experiences when I switched to using Brave browser that blocks lots of tracking. Many websites that didn't show me those captchas and Cloudflare protection layers now have started to pop up on a regular basis.
EPendragon··on I was wrong about robots.txt
You are bringing a great point. In some cases having your data as available as possible is the best thing you can do for your business. Letting them crawl and scrape creates means by which your product is found and advertised.

In other cases, like technical writing, you might want to protect the data. There is a danger that your content will be stolen and nothing will be given in return - traffic, money, references, etc.

EPendragon··on I was wrong about robots.txt
I agree. I mostly mean that it is good to strive towards a system of rules that will be easy to follow and difficult to break by default. That is an ideal case. In reality, it is never that simple.
EPendragon··on I was wrong about robots.txt
This is a really cool tool. I haven't seen it before. Thank you for sharing it!

On their README.md they state:

> This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.

I love the idea!

EPendragon··on I was wrong about robots.txt
I have not heard of humans.txt before. It is apparently used for acknowledgement and crediting the dev team who created the resource.
EPendragon··on I was wrong about robots.txt
It is so interesting to track this technology's origin back to the source. It makes sense that it would come from a background of limited resources where things would break if you overwhelm it. It didn't take much to do so.
EPendragon··on I was wrong about robots.txt
Yeah, I don't even think I 100% understand how the algorithm and moderation works here.

I will consider to bring in more content that is other than my own. I did read the guidelines for HN, and I saw that it encourages members to share what they find, and occasionally their own stuff. Appreciate your opinion!

EPendragon··on I was wrong about robots.txt
To be honest, I also think that the discussion has become more interesting than what I wrote in the article. I have learned quite a few things for myself. And on top of that it has been great to hear some feedback in regards to some thoughts that I wrote.

There are more articles available in my blog than I have shared here. I don't think that everything that I write is shareworthy on HN. There are some that I find to be more interesting. Those are the ones I end up sharing.

Like you have noticed, I try to share other interesting resources that I find online too. Is there a ratio of self/others content that would help keeping a healthy community?

EPendragon··on I was wrong about robots.txt
It is an interesting tidbit. I personally don't need Google to remove it from indexing. It is more of a "I don't care if they index it". I mostly care about the scrapping and not indexing. I do understand that these terms could be used interchangeably. In the past I might have conflated them.
EPendragon··on I was wrong about robots.txt
> Computation was still expensive

I think this is still relevant today in cases where there are not many resources available: think free tiers, smallest fixed cost/fixed allocation scenarios, etc.

EPendragon··on I was wrong about robots.txt
It definitely was (and still is to a degree) self-inflicted :)

It is fun to learn something when you discover an unintended consequence and then work backwards from it.

EPendragon··on I was wrong about robots.txt
That is what I first thought of when I started seeing more comments noting that it might be a good idea to allow for search engine traffic. All of these suggestions lead me to think about it some more. I definitely will need to understand the domain of SEO and search engine indexing a little deeper if I want to make an educated decision about it and not shoot myself in the foot in the long run.

I appreciate your suggestion!

EPendragon··on I was wrong about robots.txt
I believe that this might give birth to new tools, protocols, and tech which will enable the next evolution of the Open Web into something akin to Protected Open Web.

I very much dislike the invasive scrapping approaches. If something were to be done about it, it would result in a new way that clients interact with resources on the web.

EPendragon··on I was wrong about robots.txt
I can see how it could lead to more overhead when communicating with the requesters. That could be a lot in the event that lots of them might want to crawl your resource.

I can see the argument that if I want to hide something, I should put it behind the layer of authentication. Robots is not a substitution for proper access control mechanisms. It is more of a "if they do honor this document, this would reduce the unnecessary traffic to my site" notion.

I appreciate you highlighting your personal experience in dealing with bots! I like the ideas of monitoring and being behind something like Cloudflare tools which would protect against the major influx of traffic. I think this is especially important for smaller sites which either use low or free tiers of cloud services.

EPendragon··on I was wrong about robots.txt
Fair enough. It should be sufficient to say one way or the other.
EPendragon··on I was wrong about robots.txt
I had to discover the Post Inspector tool which was very helpful as it provided error messages.

I didn't know that about the doctype tags. I must have had them right from the beginning. Didn't encounter those issues. It good to know though.

EPendragon··on I was wrong about robots.txt
A brilliant analogy. Robots doesn't provide access controls. Authentication and authorization do.
EPendragon··on I was wrong about robots.txt
I believe that as search engines continue to move toward AI summaries and responses, it will reduce the traffic to the websites since most people will be ok accepting the answers that the AI gave them.

My approach right now is to rely on social media traffic primarily where you can engage with the readers and build trust with the audience. I don't plan on using any advertising in the near future. While that might change, I am convinced that more intentional referral traffic will generate more intentional engagement.

EPendragon··on I was wrong about robots.txt
That was the first thing that I have learnt about the robots.txt file. Even RFC 9309 Robots Exclusion Protocol document: https://www.rfc-editor.org/rfc/rfc9309.html - mentions:

> These rules are not a form of access authorization.

Meaning that these are not enforced in any way. They cannot prevent you from accessing anything really.

I think the only approach that could work in this scenario would be to find which companies disregard the robots.txt, and bring it to the attention of technical community. Practices like these could make the company look shady and untrustworthy if found out. That could be one way to keep them accountable, even though there is still no guarantee they will abide by it.

EPendragon··on I was wrong about robots.txt
Thank you for that note! I didn't know that. It is something I will need to figure out: either I'm ok with not being in the search engine OR I will update the robots.txt. Currently I'm relying on the social media traffic and the word-of-mouth marketing.
EPendragon··on I was wrong about robots.txt
One way to test if the model respects robots.txt could be to ask the model if it can scrape a given URL. One thing that it doesn't address though is the scrapping of the training data for the model. That area feels more like a wild west.
EPendragon··on I was wrong about robots.txt
There is an interesting article about AI tarpits that addresses a similar issue: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-...

The argument is basically to have them scrape your website indefinitely wasting their resources for the bots that decide to ignore your robots.txt (or any bot if you desire)

EPendragon··on I was wrong about robots.txt
I think a paragraph could have been enough to describe the issue.

My goal with this post was to describe my personal experience with the problem, research, and the solution - the overall journey. I also wanted to write it in a manner that a non-technical person would be able to follow. Hence, being more explicit in my elaborations.

EPendragon··on I was wrong about robots.txt
Hey, @jarofgreen! Thank you for the feedback!

1) I only considered LinkedIn alone since I have been posting there and here on HN, and that's it. I figured I will let it play out until I need to allow for more bots to access it. Your suggestion of other people wanting to share the links to the blog is a very valid one that I haven't thought about. I might need to allow several other platforms.

2) With Google and other search engines I have seen a trend towards the AI summaries. It seems like this is the new meta for search engines. And with that I believe it will reduce the organic traffic from those engines to the websites. So, I do not particularly feel that this is a loss for me.

I might eat my words in the future, but right now I think that social media and HN sharing is what will drive the most meaningful traffic to my blog. It is a word-of-mouth marketing, that I think is a lot more powerful than finding my blog in a Google search.

I will definitely need to go back and do some more research on this topic to make sure that I'm not shooting myself in the foot with these decisions. Comments here have been very helpful in considering other opinions and options.

EPendragon··on I was wrong about robots.txt
Honestly, I am also surprised how this got on the frontpage. This was supposed to be a small post of what I have learnt in the process of fixing my LinkedIn previews. I don't know how we got here.
EPendragon··on I was wrong about robots.txt
That's exactly it. It was one of those unintended consequences of blocking everything that led me down the road of figuring it out.

Like other commenters have indicated, I will likely need to go back and allow some other social media to access the OPG data for previews to render properly. But since I mostly post on LinkedIn and HN, I don't feel the need to go and allow all of the other media at the moment. That might change in the future.

← PreviousPage 2 of 3Next →