Hacker News Reader

Comment by yoaviram | Hacker News Reader

If you were Meta designing this chatbot, would you plant hidden instructions for it in public pages, increasing the size of the page, and leaking what is internal info for everyone to see? Is this the natural design choice? Or would you give the chatbot its own API with the same text + the relevant instructions? Does not seem to me to be a sensible design choice. We are not in the days of using hidden HTML to communicate with internal subsystems.

View on HN

Honestly, this is a clever way of doing this. It means the additional instructions are stored alongside the text they reference, which makes it easier for doc writers to keep in sync and it probably also makes it more effective since the bot is guaranteed to read it at the right time in the right context.

I doubt they actually send it with every request and hide it in CSS, it probably does some other kind of check (IP, user agent, token...). This also tracks with the fact it is now entirely gone - presumably they just fixed that if statement and it's again chatbot-only.

You're giving them much more benefit of doubt than they deserve

What? This is the most logical solution and the most likely one given the behaviour seen in the post. And even if they were sending every human reader the LLM content too, who cares? It's a few kilobytes at worst in a multi-megabyte page.

> presumably they just fixed that if statement and it's again chatbot-only.

Including the text for chat-bot only wasn’t an if statement.

If it had been an if statement, it wouldn’t be invisible to humans: it would just be visible. Period. There’s no need to display none.

Display none is for humans.

This was intended, without any need remotely to doubt it, to hide the text from humans.

Was it just incompetent implementation? Or nefarious? Who knows? Perhaps? …but there’s no universe this was ever intended to just be for chat bots.

Thats just obviously nonsense. It wasn’t a bug.

It was deliberate.

Given they probably need to go through layers of management, across teams, and go through relevant testing to adjust the LLM prompt, this was probably a nice quick and flexible way to create a "shadow" prompt.

Maybe? But it also doesn't necessarily belong into the prompt. The model probably accesses the help pages as a RAG knowledgebase, so putting it there makes sense.

It still doesn't make sense. That pages are probably built from an internal repository/database where that information is in a human readable format (e.g. Markdown or some sort of CMS) that is then processed by a CI to produce the public static HTML pages.

It would be more efficient to give the chatbot context the source files, in a format that is more efficient for that (Markdown documents, for example) and that will of course contain the internal notes, that are stripped during the build of the public HTML version.

Why a chatbot should waste resources in parsing a public webpage for the same information that you have internally in possibly a more appropriate format?

Also having a check for user agent/IP to do so is still inefficient, because you can't serve static HTML/CSS/JS and thus you have to do processing work on each request, a thing that doesn't make a lot of sense for static content of this kind.

Nobody said it's efficient, welcome to the modern IT landscape. The docs are surely some SaaS with a WYSIWYG editor, access controls and analytics. Every page probably does a dozen database lookups and involves 3 mixroservices and 4 external providers minimum.

An LLM coded it that way in a vibe session, nobody reviewed it critically, and so that's what it is.

that is exactly what they would do, otherwise it wouldn't pass any review. Unless this was generated and reviewed by AI during their experimentation phase, but I doubt it very much.

That way it also works with third-party chatbots

I think this is exactly the point the article is making...

Reply on news.ycombinator.com