In last year, some people were publishing aider /ollama/open router [1] and now thankfully people are publishing all around about pi/qwen/llama.cpp/openrouter. It's widespread.
[1] https://alexhans.github.io/posts/aider-with-open-router.html
152 karma · joined April 11, 2023
https://ai-evals.io (community site for eval-driven development as a shared language for product building)
In last year, some people were publishing aider /ollama/open router [1] and now thankfully people are publishing all around about pi/qwen/llama.cpp/openrouter. It's widespread.
[1] https://alexhans.github.io/posts/aider-with-open-router.html
- [1] https://youtu.be/Id-zzOGnN6A (Website part at 1:42 calling out the insurance example).
The point being that sometimes the tools themselves don't need to survive because you take the lessons from one thing to another (e.g. move semantics and rust/modern c++)
[1] - https://pragprog.com/titles/btlang/seven-languages-in-seven-...
Do you have further insights on AI and education since?
This applies to all software projects except for the "useful to you" part, which turns into valuable to "your/the business"
It had such a pleasant interface for that time.
But yeah, one of the things made me go for vim over Emacs a long time ago was its relationship with touch typing and not leaving the home row with the vim modes.
Learn Vim script the hard way [1], even if you didn't end up writing any actual vim script, was a game changer in terms of understanding the semantics.
I looked superficially at your site/repo and based on that initial impression:
- Your approach of comparing different parts of the "black box" which affects agent behaviour (Harness, foundation model, skills, context (in your case the loaded on AGENTS.md context) is closely aligned with how I both think and operate. - You're both tackling the "regression" and the "answer hypothesis easily" problems.
> Stalking your profile (sorry..) I see you're pretty deep in the eval space, so I'm super curious what your approach has been to being rigorous for things like skill changes?
It depends on the level of automation and risk profile. For skills I use this framework of thinking [1] and encourage evals/ground truth as soon as possible so that you can have automatic feedback loops for the markdown part and for the deterministic part (scripts). Once you have the eval/ground truth pair, you're almost doing TDD or Eval Driven Development (which is quite hard the first times you try and realize you actually need to think about intent). The scripts should definitely have their own unit tests for the "skill iteration" in the event that a mutation is desired to cover new behaviour/fix wrong behaviour.
On Agent Skills, it may seem tempting to want more "openness" for the AI to solve the problem creatively but, more often than not, you've described a repeatable workflow and you want predictability and stability instead of novelty so it's really about 1) How can I freeze it to keep being good enough as much as possible 2) How can I know if something happened somewhere which changed the black box (e.g. coding harness auto model picking screws things up 3) How can I make the skill itself ETC (Easy to change), to keep control. Local Models can be a great tool for stability in some scenarios.
In particular, I prefer pass/fail (binary) outcomes instead of scoring which doesn't help regression decisions. Defining "good enough" should be very clear. Flakiness is not a good thing to accept, if the outcomes are consequential.
Anything actually risky should be solid RBAC/policy which doesn't really depend on the LLM.
I had a site that I didn't manage to make visible in HN to create a community for ai-evals.io. I've since interacted with a few people, developed further insights and given some private talks but need to get back to publishing outfacing and trying to contact more people interested in this space because it's absolutely critical. There's a lot of nuance in how different environments think about the eval problem differently: It's all about tracing and course correcting after launch, it's about simulations, sandboxing, security, automatic eval generation, etc.
In any case, I'll try to be more present from now on, and especially from June onwards to try to exchange insights in the open with people who are exploring different solutions in this space.
[1] - https://alexhans.github.io/posts/series/evals/building-agent...
That's fair but I think this is similar to power tools like vim, obsidian or others. There's the path of grabbing other people's workflows and not being able to modify them to really tailor the tool to your needs and there's the minimal incremental path that empowers you and gives you control all the way through. It gets you to understand the tools and you'll be able to think possibilities that match your exact problems.
I'm not dogmatic about it but I do really recommend it. You can see the transformative shift once people start "skill building" instead of "skill consuming".
Edit: The approach I mention works with non engineers/developers. So there's no different technical bar.
Conceptually, you should treat them as incremental software instead of magic you grab from others [1]
The killer feature is that coding harnesses tend to have SkillBuilder agent skills so creating them becomes very easy and you can evolve them.
I recommend you build your own for your particular pain points.
Very simple example [2] showing what another user mentioned around "evals" so that you can really achieve good enough correctness for your automation.
- [1] https://alexhans.github.io/posts/series/evals/building-agent...
- [2] https://alexhans.github.io/posts/series/evals/sketch-to-text...
Where are you keeping your design? If it's a WYSIWYG doc like word, what would it take for you to switch to a more text driven world?
If you don't force yourself to see it as a scarce resource it's very easy to end up in situations where it takes up most of your time.
You're waiting a whole week for an update or to push people.
I'm a huge proponent of "async over sync" and "sync as soon as necessary". Don't wait to the next meeting in 3 days to escalate.
People whose calendar looks like meetings only and are always willing and able to throw complexity to others won't feel the pain of not having a meeting budget [1] but other roles will want to avoid eating up all their time on this type of admin.
I encourage people to explore how extremely simple Agent Skills workflows can already do a lot of the admin legwork than an executive assistant would and free their time from admin bureocracy and red tape, as much as possible. If there's no programmatic endpoint or MCP, use something like playwright. A little goes a long way.
If you can define good enough for you and local llms can meet that you'll get:
- no vendor lock-in (control)
- price
- stability (you decide when to hot swap with newer models)
- speed (control)
- full observability and predictability.
- Privacy / Data Locality (Depending on implementation of infrastructure).
- [1] https://alexhans.github.io/posts/series/evals/measure-first-...
Both are useful to different people (and role families) in different ways and if you don't feel certain pain points, you may not care about some of the value they provide.
Agent skills are useful because they're standardized prompt sharing but more than that, because they have progressive disclosure so you don't bloat your context with an inefficietly designed MCP and their UX is very well aligned such that "/SkillBuilder" skills are provided from the start and provide a good path for developers or non traditional builders to turn conversations into semi or full automation. I use this mental model to focus on the iteration pattern and incremental building [1].
[1] https://alexhans.github.io/posts/series/evals/building-agent...
No need to read them, just a vibe check would be insightful. It's weird how branding, even before AI had a lot of the same catchy patterns, and now it's hard to define what is the right prose (engineers might want one thing and other role families others) and sometimes you're trying to write almost with the "everyone else bag in mind" because that's the personal connections you link your "here's what I often repeat in written form".
For those that don't know Armin, he's the creator of flask among many things and has some cool projects like around like gondolin.
As for pi, you should absolutely try it out if you've used opencode/continue-dev or even if you've used the less open cloud vendor coding harnesses like Claude/Codex.
I do agree that they write in a way that requires context.
- [1] https://open.spotify.com/album/1R6rh9My8CTK4DqZorJR0V?si=3Ct...
If you have specific song/interpretation recommendations I'd love to hear them.
You'll define exactly what good looks like.
I wrote this to help people (not just Devs) reason about agent skills
https://alexhans.github.io/posts/series/evals/building-agent...
And this one to address the drift of non determism (but depending on the audience it might not resonate as much)
https://alexhans.github.io/posts/series/evals/error-compound...
Depending on how they're made up, different teams do vastly different things.
No evals at all, integration tests with no tooling, some use mixed observability tools like LangFuse in their CI/CD. Some other tools like arize phoenix, deepeval, braintrust, promptfoo, pydanticai throughout their development.
It's definitely an afterthought for most teams although we are starting to see increased interest.
My hope is that we can start thinking about evals as a common language for "product" across role families so I'm trying some advocacy [1] trying to keep it very simple including wrapping coding agents like Claude. Sandboxing and observability "for the masses" is still quite a hard concept but UX getting better with time.
What are you doing for yourself/teams? If not much yet, i'd recommend to just start and figure out where the friction/value is for you.
- [1] https://ai-evals.io/ (practical examples https://github.com/Alexhans/eval-ception)
Pretty much. I've been advocating this for a while. For automation you need intent, and for comparison you need measurement. Blast radius/risk profile is also important to understand how much you need to cover upfront.
The Author mentions evaluations, which in this context are often called AI evals [1] and one thing I'd love to see is those evals become a common language of actually provable user stories instead of there being a disconnect between different types of roles, e.g. a scientist, a business guy and a software developer.
The more we can speak a common language and easily write and maintain these no matter which background we have, the easier it'll be to collaborate and empower people and to move fast without losing control.
- [1] https://ai-evals.io/ (or the practical repo: https://github.com/Alexhans/eval-ception )
With people acknowledging heatwaves and energy issues, I find it interesting how that's seldom part of the conversation.
If had to make an executive choice with no further analysis at this moment I'd put them all in their respective original times and move Spain and any outlier to their proper timezone (a vertical map alignment of sorts)
People put different weights to different arguments.
For the Spain argument below. I actually think it's quite uncomfortable to be +1 and +2 in daily life because people leaving office at 5pm are actually leaving at 3pm under scorching sun. The difference of having light until 23 instead of 22 is negligible in a country that is still up at night in winter.
I can't cite anything at the moment but from what I can recall, economic benefits of switching during the year have not been as tauted and the cost of changing every year has been harmful in many ways (operational being one), but I think here the discussion is where should countries land.
I hope that a country like UK doesn't decide to switch to +1 and the same for Europe, further separating themselves from the American continent countries with the focus on summer sunlight where summer already has a huge window of sun and people often tend to want to escape that heat.
Technically, that means being able to install Linux, run local models, and use open-source software as we see fit.
Legally, it's opposing compliance guises that erode those rights, like backdoors or restrictions on what can run so that we no longer really in control of the hardware we own but need to adjust to the whims of the controller/operator, which could, at a moments notice, default to these dark patterns for "pragmatic reasons" of their own which don't align with your interests.
We know enough bad stories for the "internet of things" devices. Anyone interested in FOSS and control should probably invest in this angle.