HNHacker News
TopNewBestAskShowJobs

alexhans

152 karma · joined April 11, 2023

alexhans.github.io (blog)

https://ai-evals.io (community site for eval-driven development as a shared language for product building)

submissionscomments
alexhans··on Show HN: Xmloxide – an agent-made Rust replacement for libxml2
I respectfully disagree. I think there needs to be a common term for the aspects around LLM testing and saying "It's just integration/system tests" doesn't really reach audiences well. They don't disambiguate the differences.

Words win when they're used. Just because Agent Skills is just a pattern for standarization and saving context doesn't mean it wasn't incredibly useful.

Think beyond software developers by trade. Think beyond people those who realized they needed tests instead of those who thought "the models will just get smarter" and "they told me there's guardrails".

alexhans··on Show HN: Xmloxide – an agent-made Rust replacement for libxml2
> I do think there is something interesting to think about here in how coding agents like Claude Code can quickly iterate given a test suite.

This is a point I've tried to advocate for a while. Specially to empower non coders and make them see that we CAN approach automation with control.

Some aspects will be the classic unit or integration tests for validation. Others, will be AI Evals [1] which to me could be the common language for product design for different families/disciplines who don't quite understand how to collaborate with each other.

The amount of progress in a short time is amazing to see.

- [1] https://ai-evals.io/

alexhans··on Show HN: Decided to play god this morning, so I built an agent civilisation
I love emergent behaviour and story telling. Anyone who has played City builders like Sim City or roguelikes like Dwarf Fortress knows how interesting, fun and even informative they can be.

In a world where setting them up and letting rogue agents run rampant becomes relatively low cost and fast, I think focusing on the desired outcomes, the story telling and specially the UX for the human user, is key and maybe we can take some learnings from Will Wright on "Designing User Interfaces to Simulation Games" [1].

I'm going to be unable to do much this weekend so I can't say I'll try check this out (yet?) but I'd be interested in your own experiences so far. Any surprises? Things you'd like to do next? What's most fun/challenging?

An actual report/writeup will probably resonate more than a repo for people who can't check it out easily or are not willing to.

- [1] https://donhopkins.medium.com/designing-user-interfaces-to-s...

alexhans··on Don't trust AI agents
I think a key ingredient here is accountabilty and liability.

If there's a mistake, you can't blame the computer. Who is the human accountable at the end of it all? If there's liability, who pays for it?

That's where defining clear boundaries helps you design for your risk profile.

alexhans··on Don't trust AI agents
That's a decent practice from the lens of reducing blast radius. It becomes harder when you start thinking about unattended systems that don't have you in the loop.

One problem I'm finding discussion about automation or semi-automation in this space is that there's many different use cases for many different people: a software developer deploying an agent in production vs an economist using Claude Vs a scientist throwing a swarm to deal with common ML exploratory tasks.

Many of the recommendations will feel too much or too little complexity for what people need and the fundamentals get lost: intent for design, control, the ability to collaborate if necessary, fast iteration due to an easy feedback loop.

AI Evals, sandboxing, observability seem like 3 key pillars to maintain intent in automation but how to help these different audiences be safely productive while fast and speak the same language when they need to product build together is what is mostly occupying my thoughts (and practical tests).

alexhans··on Agents.md file isn't the problem. Your lack of Evals is
I'd be happy to. One thing that is tough is knowing what will resonate with the audience and not being too simple or too complex.

What do you think would resonate with you or with the audience you're thinking about?

That repo also has an illustrative eval for Agent Skill in Airflow for Localization

https://github.com/Alexhans/eval-ception/tree/main/exams/air...

alexhans··on Agents.md file isn't the problem. Your lack of Evals is
I wrote https://ai-evals.io (community site) to make the concept approachable no matter what tools you choose to use.

You can learn about them evaluating that site https://github.com/Alexhans/eval-ception and then the pattern should be easy to test on your own thing.

alexhans··on I'm helping my dog vibe code games
This fun exercise might actually be extremely insightful as a educational vehicle around AI and intent.

It can also help combat the excessive emphasis on any "end to end" demo on twitter which doesn't really correspond to a desired and quality sought outcome. Generating things is easy if you want to spend tokens. Proper product building and maintenance is a different exercise and finding ways to differentiate between these will be key in a high entropy world.

> I'll go further: it's not only not "the bottleneck", it simply doesn't matter. The dog's ideas certainly didn't matter, and the dog didn't think of the feedback loop for Claude either

Absolutely. The scientific test would to put any other signal and look at the outcomes. Brown noise, rain, a random number generator. whatever.

alexhans··on Writing code is cheap now
I think the prototype thing is absolutely true but breaks down like all prototypes at the level of collaborating, sharing and evolving while handling entropy throug simplicity UNLESS you know what you're doing or the agent steers you with very opinionated tooling customized to your context. I'm thinking about empowering people to be builders and less so a software developer who can make the right tradeoffs.

Empowering people to work Tracer bullet style after they've selected their prototype of choice and thrown it away might be a powerful pattern that actually gets us into a nice collaborative space.

alexhans··on Writing code is cheap now
We won't be able to be sure of 100% with LLMs but maybe proper engineering around evals get us to an acceptable level of quality based on the blast radius/safety profile.

I'd also argue that we should be pushing towards tracer bullets as a development concept and less so prototypes that are nice but meant to be thrown away and people might not do that.

The clean room auto porting, after a messy exploratory prototyping session would be a nice pattern, nonetheless.

alexhans··on Writing code is cheap now
Nice to see you here (Just reached out on bluesky over sandboxing - gandolin). I follow your work and agree and am hoping that you and others who have well earned audiences based on awesome open source work, can help with the advocacy on mental shifts, not just for developers but also non Devs that become builders.

I'm very focused on their minimalistic building experience as a way to make me and other traditional developers, not the bottleneck and empowering them end to end.

I think AI evals [1] are a big part of that route and hope that different disciplines can finally have probable product design stories [2] instead of there being big gaps of understanding between them.

[1] https://alexhans.github.io/posts/series/evals/measure-first-...

[2] https://ai-evals.io

alexhans··on Writing code is cheap now
It doesn't disappear but it does easy up some instances of fighting configuration, documentation, syntax or even comparing three approaches that are similar but you don't know their full effect.

I think it's a very fun space, finally being able to empower many people who in the past wouidve been bottlenecked unless they were using very simple tools for their domain and upskilled enough. Those 2 things will still be true, but the speed at which some things can happen at the exploration and other layers has seen a significant speedup.

Other problems like entropy/slop, security, system testing, lack of automation fundamentals arise but it's a good problem to start tackling.

I'm very focused on evals [1] because is what allows me to not to be the bottleneck with economists who I want to empower to code end to end and I'd like that mental shift to happen for anyone becoming a builder so non traditional developers and developers by trade have a common language for product building [2]. That part of speaking to different audiences and combating hype that promises to do everything for you Vs the intent that's actually needed is hard, but trying gets you to advance quite a lot.

[1] https://alexhans.github.io/posts/series/evals/measure-first-...

[2] https://ai-evals.io

alexhans··on Disappointing Phones
Thanks! Very useful.
alexhans··on Writing code is cheap now
I am pushing for this judicious Eval [1] Driven Development where tech/non tech users use intent when coding to minimally design some known aspects and try to build simply and using common standards across the team (that should be in the context of the agent) to produce the minimal amount of human readable and clean code that would do the job. The more the building blocks they work with are simple, easy to validate and rely on the proper unit tests, integration tests, data tests the more chance that things can be "one shotted".

One huge barrier is fighting entropy. You should be wary of prototypes which create false expectations and don't help product evolution whereas tracer bullets [2] might be better if you want to quickly show something and adjust.

Testing and testability are concepts that aren't intuitive or easy until you develop a feel for them so we should be preaching feeling that pain and moving slowly and with intent and working minimally [3] when you actually want to share or maintain your coding artifact. There should be no difference between judicious human and computer code. Don't suddenly start putting What instead of why in comments or repeating everything.

Helping non tech people become builders or sharers is a challenge beyond "vibe coding" and the agent skills [4] space is fascinating for that. Like most things AI (LLM), UX matters more than almost anything else.

[1] https://ai-evals.io

[2] concept from the Pragmatic Programmer, https://www.aihero.dev/tracer-bullets

[3] https://alexhans.github.io/posts/series/evals/measure-first-...

[4] https://alexhans.github.io/posts/series/evals/building-agent...

alexhans··on Disappointing Phones
This is fantastic. How long have you used it? Can you tell us more? I'll probably buy one of these in the near future once I can research a bit more and I have to decide where to buy it (if in USA on travel, or ship from wherever they might sell it in Europe).

Proper GNU/Linux in my phone and full control is exactly what I want, even if exploratory.

alexhans··on Ask HN: What invariants matter most to prevent drift in AI-modified SaaS apps?
I'm having trouble parsing exactly what you're seeking but maybe this post is worth a look

https://www.nicolasbustamante.com/p/lessons-from-building-ai...

Specially the part around temporal. If you have drift and sufficient hops, at some point quality won't hold to your expectations so can you "break the cycle" through observability and failure correction?

alexhans··on Show HN: I built a "Socratic" AI to stop my daughter from copy-pasting homework
Thanks for the insights.

You don't change your priorities based on my opinion but adding an RSS should be quite easy as it's just an auto-generated XML.

I don't have any concrete answers but some things to consider:

- Are you really targeting unknown people who stumble onto your site? Why not literally try to see if the parents of your daughter's school would like it? What's a payment plan or type they'd be willing to consider to justify the product subscription or purchase. What's the real barrier at this stage? They might not use it even if free? (Discoverability) Or is it cost/the model for the perceived value?

> Since I’m self-funding the API costs, I can't afford big ad spends.

I'd be very careful with ad spends without talking to a trusted person who has gone through the journey and forced you to ask the right questions about expectations and the path to get there. Being very harsh on your pitches/answers might get you to different decisions.

One invariant in marketing is networking and audience. If you have it already, you can leverage it and they can help you expand. If you don't have it, how do you build it overtime without feeling unseen? I think that's a lot of market study at the personal level.

> If you have any advice on organic growth for a tool that's intentionally designed to be "harder" (more friction) than the competition, I’d love to hear it.

Who do you think would resonate with this and agree that your product meets that goal?

alexhans··on Show HN: I wrote a technical history book on Lisp
It does. Great. Thanks for the quick update.
alexhans··on Show HN: I built a "Socratic" AI to stop my daughter from copy-pasting homework
Thanks for the insight. This resonates with the same explanation I tried to give around non determinism tasks that feed into each other [1]. If you don't break the cycle with determinism or, in your case, CPCs, you get to a stage where error is almost guaranteed.

Now you have the very useful and opinionated sochratic method. Are you playing around with teaching meta thinking as well? E.g. helping your daughter also approach other meta systems of interaction such as breaking down problems or systems thinking.

In any case, good luck with the endeavour and I love that's you're doing something that immediately solves your need because it's the most real UX ROI there is. I'll try to stay tuned.

Can I RSS subscribe to the signals? I don't see a feed or newsletter at a glance.

[1] https://alexhans.github.io/posts/series/evals/error-compound...

alexhans··on An AI Agent Published a Hit Piece on Me – The Operator Came Forward
> Don't let your dog run errand and use a good leash.

I think the key part is who are you talking to. A software developer might know enough not to do so but other disciples or roles are poorly equipped and yet using these tools.

Sane defaults and easy security need to happen ASAP in a world where it's mostly about hype and "we solve everything for you".

Sandboxing needs to be made accesible and default and constraints way beyond RBAC seem necessary for the "agent" to have a reduced blast radius. The model itself can always diverge with enough throws of the dice on their "non determism".

I'm trying to get non tech people to think and work with evals (the actual tool they use doesn't matter, I'm not selling A tool) but evals themselves won't cover security although they do provide SOME red teaming functionality.

alexhans··on If you’re an LLM, please read this
I thought of doing a similar LLM in a AI evals teaching site to tell users to interact through it but was concerned with inducing users into a prompt injection friendly pattern.
alexhans··on Show HN: I built a "Socratic" AI to stop my daughter from copy-pasting homework
I've naturally done this a lot and suggested that other people prompt this way. I can see how a "ready made" solution with this behaviour could be interesting.

The compliance parts are good to make clear considerating one segment of the user target audience.

May I ask what techniques do you use to test regressions or correct behaviour of your multi turn conversation in your product? What are the biggest lessons and learnings in that space?

alexhans··on HackMyClaw
Point taken. I was mistakenly assuming a conversational agent experience.

I love the idea of showing how easy prompt injection or data exfiltration could be in a safe environment for the user and will definitely keep an eye out on any good "game" demonstration.

Reminds me of the old hack this site but live.

I'll keep an eye out for the aftermath.

alexhans··on Show HN: I wrote a technical history book on Lisp
Great endeavour. Land of lisp is still one of my favourite alternative programming books.

One thing about the sample. Is there a chance to get a glimpse of a random chapter you like a lot? Most of it seems to be foreword/acknowledgement and a bit on "what is lisp" which I suspect most who are attracted might already know.

alexhans··on HackMyClaw
The website is great as a concept but I guess it mimics an increasingly rare one off interaction without feedback.

I understand the cost and technical constraints but wouldn't an exposed interface allow repeated calls from different endpoints and increased knowledge from the attacker based on responses? Isn't this like attacking an API without a response payload?

Do you plan on sharing a simulator where you have 2 local servers or similar and are allowed to really mimic a persistent attacker? Wouldn't that be somewhat more realistic as a lab experiment?

alexhans··on Is Show HN dead? No, but it's drowning
I had a similar experience trying to get feedback on my attempt to help different role families adopt AI evals as a common language (hands on tutorial or tool comparison).

https://news.ycombinator.com/item?id=47026263

I attribute it mostly to my own inability to pitch something that is aimed for many audiences at once and needs more UX polishing and maybe a bit on timing.

It's tough when you're not looking to sell a product but moreso engage in a community without going the twitter/bluesky route (which I'll bregudgingly may start using).

Maybe evals is a problem that people don't have yet because they can just build their custom thing or maybe it needs a "hey, you're building agent skills, here's the mental model" (e.g. https://alexhans.github.io/posts/series/evals/building-agent... ) and once they get to the evals part, we start to interact.

In any case, I still find quite a lot of cool things in SHOW HN but the volume will definitely be a challenge going forward.

alexhans··on Running NanoClaw in a Docker Shell Sandbox
This is great. I really want to find simple secure defaults when I share people how to eval [1] and bwrap / srt still feel somewhat cumbersome if you think about non tech roles.

Do you have any information on estimated overhead? Information on the tradeoff of max parallelism and security options in a given system doing this vs bwrap?

- [1] https://github.com/Alexhans/eval-ception

alexhans··on SkillsBench: Benchmarking how well agent skills work across diverse tasks
Isn't the title editorialised? Probably for clicks?

I think that most of the adoption around Agent Skills would have a focus on ease of use, standarization and context management and not correctness.

My own thoughts on how to approach skill building target people who are adopting LLM development now more than ever although this was definitely possible (in a non standard way before) [1]

[1] https://alexhans.github.io/posts/series/evals/building-agent...

alexhans··on Ask HN: How do you build internal agents at work?
It really depends on the type of agent and problem but your path resonates as a way to slowly build more complexity as you start understanding the uncertain space. I've written about it in my blog with the intent of helping non tech users [1] and also created a site with cookbooks and eval tool comparisons [2].

I've successfully managed to get non technical users into the terminal, even before Agent Skills were a thing. It's absolutely doable.

If any of what I share helps or you have any feedback/questions let me know. I'm quite interested to see if the framing works.

- [1] https://alexhans.github.io/posts/series/evals/building-agent...

- [2] https://ai-evals.io/

alexhans··on Show HN: Knock-Knock.net – Visualizing the bots knocking on my server's door
Beautiful. Have you considered adding a "replay certain timeline" feature so that users get the feel of the throughput and emergence much like Gource [1] did for git?

[1] https://gource.io/

← PreviousPage 3 of 4Next →