14 karma · joined June 25, 2026
You can achieve this by moving from summarize to deconstruct. For example, standard agents default to generic summaries. You must force the agent to act like a senior engineer conducting a code review or a research lead. You can do this by prompting for architectural boundaries and friction points, not just descriptions, by asking "why", not "what".
You could also use tactical prompts for Code onboarding. An example prompt would be, "trace a single user request from the entry point API down to the database. List every module it touches and where mutations happen."
This is how I would do it if I were in your shoes, but there are certainly other methods that could be better suited for you. Either way, I hope this was at least useful in some way.
For instance, people running 32+ GPU setups probably correlate timestamps rather than trying to preserve strict per-request attribution at the rack level. This will enable these individuals to have rack/PDU power sampled every second.
Either way, I haven't seen many people publish how they instrument this in practice so take what I wrote with a gran of salt. I simple wanted to share a little bit of what I understand and I hope it helps.
I built a database website a few months ago and submitted it to Google, Bing, and Yandex. 2 months later, according to my Cloudflare dashboard, I have 1.5 million unique visitors monthly. I found that human visitors only accounted for about 10% of the total, followed by search engine crawl bots and then AI crawl bots. I also discovered that AI bots (like GPTBot, ClaudeBot, or PerplexityBot) scraped a lot rapidly without adhering to traditional crawl limits or deeply checking robots.txt files, resulting in high server loads.
That should also answer the second question, which is that AI retrieval systems index semantic structure faster than they index page content. You have to understand that AI doesn't just index your website like regular crawl bots, which index mostly your content, schema, and so on. AI bots go deeper by trying to understand your website structure, as this will also help in training other AI models.