Ask HN: How do you evaluate the skills your AI agents use?
Do you systematically evaluate agent skills, either before or while using them? If so, what tools or methods do you use? Or do you mostly rely on reading the SKILL.md and trying things out?
They match the requirements for systems I'm building and/or responsible to maintain.
So far so good.
I did this approach because most skills I come across are either too vague, too detailed, or otherwise not suitable for what I'm building. Even if I use frameworks or well-structured libraries, my system still has unique requirements.
To test, I have integration tests, evals, and also "Quality Control" style reviews.
1. Tell your agent to eval the skill itself (https://code.claude.com/docs/en/plugin-evals). This is easier and faster, but less accurate, and harder to extend to capture things like code quality.
2. Build your own benchmark out of work your normally do, and then eval the skill on that dataset. This is harder and more annoying (because creating benchmarks is hard) but provides deeper insight into how the skill affects the agent in known scenarios.
It also depends on what the skill is doing. If it's creating an artifact (like a report, easier to eval) then your strategy is very different than if the skill is changing agent behavior (like telling it to use cheap subagents under XYZ conditions, harder to eval).
The importance of doing this grows with the number of people using the skill. Working on my own projects, I don't usually bother. Working in an enterprise monorepo, suddenly the risk of shipping a bad skill is amplified, and evals become much more attractive.
Also note that skill behavior changes across different models. A skill built for Opus 4.8 may not be necessary on Opus 5.5. A skill built for Opus 5.5 might cause Haiku 5.5 to behave erratically.
TLDR: tell your agent to do it, and think about building your own.
This has bitten me so badly SO many times. Every time there's a new model, it's time to re-evaluate the skills. It got so bad I put a lot of the skill behind a CLI.
Evals saved me, but I really wish skills had a `writtenFor` field in their frontmatter so an agent could warn you it was running a skill written for a different model.
My 2c as someone who is working on this problem is that you can follow the 80/20 rule when making them for personal use - put in the 20% and you can make a good enough benchmark relatively easily. The remaining work is certainly valuable, and necessary to do if the benchmark is consequential, but it's a pain in the ass and hard to do correctly, so if you don't have to, don't worry about it