HNHacker News
TopNewBestAskShowJobs

alexhans

152 karma · joined April 11, 2023

alexhans.github.io (blog)

https://ai-evals.io (community site for eval-driven development as a shared language for product building)

submissionscomments
alexhans··on RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8
I think it's important to be able to do both so you can stay in control of the price to value created relationship.

In last year, some people were publishing aider /ollama/open router [1] and now thankfully people are publishing all around about pi/qwen/llama.cpp/openrouter. It's widespread.

[1] https://alexhans.github.io/posts/aider-with-open-router.html

alexhans··on Ryanair dark UX patterns summer 2026 refresher
The Foil Arms & Hog 12 year old skit [1] is still relevant.

- [1] https://youtu.be/Id-zzOGnN6A (Website part at 1:42 calling out the insurance example).

alexhans··on Learn SQL Once, Use It for 30 Years
I was a fan of Seven Languages in Seven Weeks [1] because it exposed you to different paradigms which you could then try to apply where they made sense on whatever tools you were using or building: prototype based, fault tolerante, funcional, logical. Very fun book when used right.

The point being that sometimes the tools themselves don't need to survive because you take the lessons from one thing to another (e.g. move semantics and rust/modern c++)

[1] - https://pragprog.com/titles/btlang/seven-languages-in-seven-...

alexhans··on AI Agent Guidelines for CS336 at Stanford
Congrats. This seems like a great prompt to ensure a useful default experience. People should not confuse this with "anti cheating" and instead helping people learn how to learn.

Do you have further insights on AI and education since?

alexhans··on The solution might be cancelling my AI subscription
A good tip for side projects is to build stuff that is actually useful to you and to make them ZeroOps (no maintenance required). If you find yourself waiting to add a feature and it's not easy to change (ETC) [1] then it's a good moment to stop and how did you lose control of the software.

This applies to all software projects except for the "useful to you" part, which turns into valuable to "your/the business"

[1] https://youtu.be/c8AzqMr87gQ

alexhans··on Time to talk about my writerdeck
When I saw this post I immediately thought of WordPerfect.

It had such a pleasant interface for that time.

But yeah, one of the things made me go for vim over Emacs a long time ago was its relationship with touch typing and not leaving the home row with the vim modes.

Learn Vim script the hard way [1], even if you didn't end up writing any actual vim script, was a game changer in terms of understanding the semantics.

[1] - https://learnvimscriptthehardway.stevelosh.com/ was

alexhans··on A Claude Code and Codex Skill for Deliberate Skill Development
Hey, sorry for the late reply.

I looked superficially at your site/repo and based on that initial impression:

- Your approach of comparing different parts of the "black box" which affects agent behaviour (Harness, foundation model, skills, context (in your case the loaded on AGENTS.md context) is closely aligned with how I both think and operate. - You're both tackling the "regression" and the "answer hypothesis easily" problems.

> Stalking your profile (sorry..) I see you're pretty deep in the eval space, so I'm super curious what your approach has been to being rigorous for things like skill changes?

It depends on the level of automation and risk profile. For skills I use this framework of thinking [1] and encourage evals/ground truth as soon as possible so that you can have automatic feedback loops for the markdown part and for the deterministic part (scripts). Once you have the eval/ground truth pair, you're almost doing TDD or Eval Driven Development (which is quite hard the first times you try and realize you actually need to think about intent). The scripts should definitely have their own unit tests for the "skill iteration" in the event that a mutation is desired to cover new behaviour/fix wrong behaviour.

On Agent Skills, it may seem tempting to want more "openness" for the AI to solve the problem creatively but, more often than not, you've described a repeatable workflow and you want predictability and stability instead of novelty so it's really about 1) How can I freeze it to keep being good enough as much as possible 2) How can I know if something happened somewhere which changed the black box (e.g. coding harness auto model picking screws things up 3) How can I make the skill itself ETC (Easy to change), to keep control. Local Models can be a great tool for stability in some scenarios.

In particular, I prefer pass/fail (binary) outcomes instead of scoring which doesn't help regression decisions. Defining "good enough" should be very clear. Flakiness is not a good thing to accept, if the outcomes are consequential.

Anything actually risky should be solid RBAC/policy which doesn't really depend on the LLM.

I had a site that I didn't manage to make visible in HN to create a community for ai-evals.io. I've since interacted with a few people, developed further insights and given some private talks but need to get back to publishing outfacing and trying to contact more people interested in this space because it's absolutely critical. There's a lot of nuance in how different environments think about the eval problem differently: It's all about tracing and course correcting after launch, it's about simulations, sandboxing, security, automatic eval generation, etc.

In any case, I'll try to be more present from now on, and especially from June onwards to try to exchange insights in the open with people who are exploring different solutions in this space.

[1] - https://alexhans.github.io/posts/series/evals/building-agent...

alexhans··on A Claude Code and Codex Skill for Deliberate Skill Development
Sorry. I'm not sure about the specific part you don't agree with. You prefer people to just use skills instead of building them?

That's fair but I think this is similar to power tools like vim, obsidian or others. There's the path of grabbing other people's workflows and not being able to modify them to really tailor the tool to your needs and there's the minimal incremental path that empowers you and gives you control all the way through. It gets you to understand the tools and you'll be able to think possibilities that match your exact problems.

I'm not dogmatic about it but I do really recommend it. You can see the transformative shift once people start "skill building" instead of "skill consuming".

Edit: The approach I mention works with non engineers/developers. So there's no different technical bar.

alexhans··on A Claude Code and Codex Skill for Deliberate Skill Development
Hey, it's awesome that you mention evals. May I ask what you currently use, or look for? Do you roll your own or use an existing framework?
alexhans··on A Claude Code and Codex Skill for Deliberate Skill Development
Skills are just a good standard to describe repeatable workflows saving context through progressive disclosure, prompt sharing and, very underused feature, also bound the non deterministic parts with determism (which could be scripts).

Conceptually, you should treat them as incremental software instead of magic you grab from others [1]

The killer feature is that coding harnesses tend to have SkillBuilder agent skills so creating them becomes very easy and you can evolve them.

I recommend you build your own for your particular pain points.

Very simple example [2] showing what another user mentioned around "evals" so that you can really achieve good enough correctness for your automation.

- [1] https://alexhans.github.io/posts/series/evals/building-agent...

- [2] https://alexhans.github.io/posts/series/evals/sketch-to-text...

alexhans··on What we lost the last time code got cheap
To me, design files living in the repo are what captures these high and low level decisions. They should express intent which is paramount for automation.

Where are you keeping your design? If it's a WYSIWYG doc like word, what would it take for you to switch to a more text driven world?

alexhans··on Set a Meeting Budget
I saw https://news.ycombinator.com/item?id=47906942 and thought that the competing perspective of meetings as a resource that is inherently wasteful deserved dedicated discussion, only justified in cases where synchronous coordination is necessary.

If you don't force yourself to see it as a scarce resource it's very easy to end up in situations where it takes up most of your time.

alexhans··on Meetings are forcing functions
Sync meetings can help some people feel pressure but it's an extremely inefficient mechanism.

You're waiting a whole week for an update or to push people.

I'm a huge proponent of "async over sync" and "sync as soon as necessary". Don't wait to the next meeting in 3 days to escalate.

People whose calendar looks like meetings only and are always willing and able to throw complexity to others won't feel the pain of not having a meeting budget [1] but other roles will want to avoid eating up all their time on this type of admin.

I encourage people to explore how extremely simple Agent Skills workflows can already do a lot of the admin legwork than an executive assistant would and free their time from admin bureocracy and red tape, as much as possible. If there's no programmatic endpoint or MCP, use something like playwright. A little goes a long way.

- [1] https://alexhans.github.io/posts/meeting-budget.html

alexhans··on Claude Code Routines
This is what AI evals [1] and local llms should be a focus of your investment.

If you can define good enough for you and local llms can meet that you'll get:

- no vendor lock-in (control)

- price

- stability (you decide when to hot swap with newer models)

- speed (control)

- full observability and predictability.

- Privacy / Data Locality (Depending on implementation of infrastructure).

- [1] https://alexhans.github.io/posts/series/evals/measure-first-...

alexhans··on Make Tmux Pretty and Usable
Tmux conflicted with my vim bindings and i tweaked the tmux.conf to fit my needs. It's worth playing around on pure tmux and figuring out what would work for you.
alexhans··on I still prefer MCP over skills
This frames MCP vs Skills as an either/or, but they operate at different layers. MCP exposes capabilities and Skills may shape how capabilities are used.

Both are useful to different people (and role families) in different ways and if you don't feel certain pain points, you may not care about some of the value they provide.

Agent skills are useful because they're standardized prompt sharing but more than that, because they have progressive disclosure so you don't bloat your context with an inefficietly designed MCP and their UX is very well aligned such that "/SkillBuilder" skills are provided from the start and provide a good path for developers or non traditional builders to turn conversations into semi or full automation. I use this mental model to focus on the iteration pattern and incremental building [1].

[1] https://alexhans.github.io/posts/series/evals/building-agent...

alexhans··on Git commands I run before reading any code
I'd be curious to see if my blog posts/titles feel like AI slop to you.

https://alexhans.github.io/

No need to read them, just a vibe check would be insightful. It's weird how branding, even before AI had a lot of the same catchy patterns, and now it's hard to define what is the right prose (engineers might want one thing and other role families others) and sometimes you're trying to write almost with the "everyone else bag in mind" because that's the personal connections you link your "here's what I often repeat in written form".

alexhans··on I've sold out
Having followed a bit of their work this makes absolute sense and I'm excited to see what it enables.

For those that don't know Armin, he's the creator of flask among many things and has some cool projects like around like gondolin.

As for pi, you should absolutely try it out if you've used opencode/continue-dev or even if you've used the less open cloud vendor coding harnesses like Claude/Codex.

I do agree that they write in a way that requires context.

alexhans··on Music for Programming
Thanks. This Is awesome
alexhans··on Music for Programming
Like others have said, for specific types of activity, I'll prefer no vocals or maybe even no music, but if vocals are fine Abba does have a great flow to it. I used to run to Abba too, at times, because it feels upbeat/positive with good enough tempo. Super trouper, for instance, makes for a great booster.
alexhans··on Music for Programming
I vary a lot but when I do classical music Mozart has occupied quite a lot of my stats, in particular a clarinet concerto by Katherine Lucy [1] and also things like Beethoven's 6th (pastoral, it's beautifully featured in Fantasia) or Grieg's morning mood.

- [1] https://open.spotify.com/album/1R6rh9My8CTK4DqZorJR0V?si=3Ct...

If you have specific song/interpretation recommendations I'd love to hear them.

alexhans··on Schedule tasks on the web
Why not set your own evals and something like pi-mono for that? https://github.com/badlogic/pi-mono/

You'll define exactly what good looks like.

alexhans··on Schedule tasks on the web
I'd say that's almost fine if they can start expressing intent correctly and thinking what good looks like. They (or some automated thing if you're building "think for them" type of products instead of "give them tools and teach them to think how to use them") can then freeze determism more and more were useful

I wrote this to help people (not just Devs) reason about agent skills

https://alexhans.github.io/posts/series/evals/building-agent...

And this one to address the drift of non determism (but depending on the audience it might not resonate as much)

https://alexhans.github.io/posts/series/evals/error-compound...

alexhans··on Launch HN: Captain (YC W26) – Automated RAG for Files
Congrats on the launch. Very quick feedback on the site, since I usually try to check out the blog section, https://www.runcaptain.com/blog layout is broken in mobile, I tried brave/chrome/Firefox.
alexhans··on Ask HN: How are people doing AI evals these days?
Very, very heterogenous and fast moving space.

Depending on how they're made up, different teams do vastly different things.

No evals at all, integration tests with no tooling, some use mixed observability tools like LangFuse in their CI/CD. Some other tools like arize phoenix, deepeval, braintrust, promptfoo, pydanticai throughout their development.

It's definitely an afterthought for most teams although we are starting to see increased interest.

My hope is that we can start thinking about evals as a common language for "product" across role families so I'm trying some advocacy [1] trying to keep it very simple including wrapping coding agents like Claude. Sandboxing and observability "for the masses" is still quite a hard concept but UX getting better with time.

What are you doing for yourself/teams? If not much yet, i'd recommend to just start and figure out where the friction/value is for you.

- [1] https://ai-evals.io/ (practical examples https://github.com/Alexhans/eval-ception)

alexhans··on LLMs work best when the user defines their acceptance criteria first
> The vibes are not enough. Define what correct means. Then measure.

Pretty much. I've been advocating this for a while. For automation you need intent, and for comparison you need measurement. Blast radius/risk profile is also important to understand how much you need to cover upfront.

The Author mentions evaluations, which in this context are often called AI evals [1] and one thing I'd love to see is those evals become a common language of actually provable user stories instead of there being a disconnect between different types of roles, e.g. a scientist, a business guy and a software developer.

The more we can speak a common language and easily write and maintain these no matter which background we have, the easier it'll be to collaborate and empower people and to move fast without losing control.

- [1] https://ai-evals.io/ (or the practical repo: https://github.com/Alexhans/eval-ception )

alexhans··on British Columbia is permanently adopting daylight time
Summers in Spain, e.g. Madrid can be extremely hot. Having the sun not set until very late can create an unpleasant city life experience.

With people acknowledging heatwaves and energy issues, I find it interesting how that's seldom part of the conversation.

alexhans··on British Columbia is permanently adopting daylight time
I don't like switching to daylight saving and back but I'd rather have that that permanently moving to +1. Then you have extreme examples which are already shifted like Spain (for historical reasons around aligning with Germany) and I don't find that alignment useful economically, in city life practice and more.

If had to make an executive choice with no further analysis at this moment I'd put them all in their respective original times and move Spain and any outlier to their proper timezone (a vertical map alignment of sorts)

alexhans··on British Columbia is permanently adopting daylight time
Don't get discouraged by being in the minority in one particular forum, specially when specific angles dominate.

People put different weights to different arguments.

For the Spain argument below. I actually think it's quite uncomfortable to be +1 and +2 in daily life because people leaving office at 5pm are actually leaving at 3pm under scorching sun. The difference of having light until 23 instead of 22 is negligible in a country that is still up at night in winter.

I can't cite anything at the moment but from what I can recall, economic benefits of switching during the year have not been as tauted and the cost of changing every year has been harmful in many ways (operational being one), but I think here the discussion is where should countries land.

I hope that a country like UK doesn't decide to switch to +1 and the same for Europe, further separating themselves from the American continent countries with the focus on summer sunlight where summer already has a huge window of sun and people often tend to want to escape that heat.

alexhans··on I built a demo of what AI chat will look like when it's “free” and ad-supported
Dark patterns degrade our computing experience and are worth illustrating, but there's a larger discussion to be had about keeping individual control over our own devices.

Technically, that means being able to install Linux, run local models, and use open-source software as we see fit.

Legally, it's opposing compliance guises that erode those rights, like backdoors or restrictions on what can run so that we no longer really in control of the hardware we own but need to adjust to the whims of the controller/operator, which could, at a moments notice, default to these dark patterns for "pragmatic reasons" of their own which don't align with your interests.

We know enough bad stories for the "internet of things" devices. Anyone interested in FOSS and control should probably invest in this angle.

← PreviousPage 2 of 4Next →