HNHacker News
TopNewBestAskShowJobs

alexhans

152 karma · joined April 11, 2023

alexhans.github.io (blog)

https://ai-evals.io (community site for eval-driven development as a shared language for product building)

submissionscomments
alexhans··on Evolving Git for the Next Decade
Fair enough. I think I'm trying to say the same thing you're saying. It's tough to navigate that world without a lot of investment in understanding the internals which, most people wouldn't do (rational behaviour for "yet another tool", if you don't come from a software development background).
alexhans··on Expensively Quadratic: The LLM Agent Cost Curve
Nice article. I think a key part of the conversation is getting people to start thinking in terms of evals [1] and observability but it's been quite tough to combat the hype of "but X magic product just solves what you mentioned as a concern for you".

You'd think cost is an easy talking point to help people care but the starting points for people are so heterogeneous that it's tough to show them they can take control of this measurement themselves.

I say the latter because the article is a point in time and if they didn't have a recurrent observation around this, some aspects may radically change depending on the black box implementations of the integrations they depend on (or even the pricing strategies).

[1] https://ai-evals.io/

alexhans··on Evolving Git for the Next Decade
Git is a beautiful piece of software but it does expose complexity in a very by programmers for programmers kind of way.

I've successfully gotten many non tech roles to use git but there's usually a lot missed in the nuances and power that is in their reach, but not quite adopted.

The learn git branching site/game [1] has always been an awesome resource but you'd like something like that UX be almost part of the initial usage. Intuitive defaults, progressive learning at the right times, etc.

Nowadays, if you can get your users to use an agent CLI like Claude/Codex/OpenCode then it's easier to have that last experience but the more git itself uses accesible abstractions the easier it should be.

[1] - https://learngitbranching.js.org/?locale=en_US

alexhans··on Claude Code: connect to a local model when your quota runs out
I just shared this in HN https://news.ycombinator.com/item?id=47026263 to see if it's possible to scale the knowledge sharing and simple and good practices which keep people in control.

It may or may not address the practical examples you need but I'd been to hear your thoughts and maybe it's possible to come up with a more illustrative one.

I didn't go for bubblewrap or similar containers yet because I didn't want to lose a specific type of baseline newcomer yet (Economists who do some coding) but I will be adding to it with whatever most elegant approaches I can find that don't leak too much complexity for things like sandboxing, system testing, integration mocking (reverse proxying), Observing with Openteleletry or otherwise, presenting benchmarks, etc.

alexhans··on Ask HN: What would you recommend a vibe coder learn about how all this works?
Congrats on the journey and desire to learn.

There's different books or approaches to learn more depending on your needs.

It's usually useful to tailor things to whatever you already produce for others or consume for yourself because then you get the most of your feedback loop.

Can you share more about what you've tried so far, with what tools, or what feels the most fun or useful ?

Depending on your answers one might recommend things like

- https://automatetheboringstuff.com/

- land of lisp (comics and games with functional programming to intentionally separate you from vibe coding traditional tools)

- web related dev with different viz or presentation tools

useful concepts regardless will be, in my mind, unit testing and how to approach problems and breaking them down but it all depends on where you want to go.

alexhans··on An AI agent published a hit piece on me
To address the issues of an automated entity functioning as a detractor? I don't think I can answer that specifically. I can brainstorm on the some of the dimensions the book talks about:

- societal norm/moral pressure shouldn't apply (adversarial actor)

- reputational pressure has an interesting angle to it if you think of it as trust scoring in descentralized or centralised networks.

- institutional pressure can't work if you can't tie back to the root (it may be unfeasible to do so or the costs may outweight the benefits)

- Security doesn't quite work the way we think about it nowadays because this is not an "undesired access of a computer system" but a subjectively bad use of rapid opinion generation.

alexhans··on GPT‑5.3‑Codex‑Spark
When I saw Spark my mind went to Apache Spark and wondered if we were learning all the lessons in orchestration of driver/worker and data shuffling from that space.
alexhans··on An AI agent published a hit piece on me
This is such a powerful piece and moment because it shows an example of what most of us knew could happen at some point and we can start talking about how to really tackle things.

Reminds me a lot of liars and outliars [1] and how society can't function without trust and almost 0 cost automation can fundamentally break that.

It's not all doom and gloom. Crisises can't change paradigms if technologists do tackle them instead of pretending they can be regulated out of existence

- [1] https://en.wikipedia.org/wiki/Liars_and_Outliers

On another note, I've been working a lot in relation to Evals as way to keep control but this is orthogonal. This is adversarial/rogue automation and it's out of your control from the start.

alexhans··on Claude Code: connect to a local model when your quota runs out
High level, the approach is:

- I'm pain point driven:

  - I can't compare what I can't measure. 

  - I can't trust to run this "AI" tool to run on its own
- That's automation, which is about intentionality (can I describe what I want?) and risk profile understanding (What's the blast radius/worst that could happen)

Then I treat it as if it was an Integration Test/Test Driven Development exercise of sorts.

- I don't start designing an entire cloud infrastructure.

- I make sure the "agent" is living in the location where the users actually live so that it can be the equivalent of an extra paid set of hands.

- I ask questions or replicate user stories and use deterministic tests wherever I can. Don't just go for LLMaaJ. What's the simplest thing you can think of?

- The important thing is rapid iteration and control. Just like in a unit testing scenario it's not about just writing a 100 tests but the ones that qualitatively allow you to move as fast as possible.

- At this stage where the space is moving so fast and we're learning so much, don't assume or try to over-optimize places that don't hurt and instead think about minimalism, ease of change, parameterization and ease of comparison with other components that form "the black box" and with itself.

- Once you have the benchmarks that you want, you can decide things like pick the cheapest model/agent configuration that does the job within the acceptable timeframe.

Happy to go deeper on these. I have some practical/runnable samples/text I can share on the topic after the weekend. I'll drop a link here when it's ready

alexhans··on OpenClaw: When AI Agents Get Full System Access. Security nightmare?
Yeah, the idea is clear. You're "integrating early" and "failing fast" and once you've understood enough about the problem you can design and optimize the right custom tool to make it more accurate, consistent, cost-effective.

To be fair, it's a micro approach of the way to approach projects rapidly where instead of trying to design too much upfront, identify what are the real value producing goals, the risks in the middle that you can foresee and get hands on in a time-boxed manner to de-risk the individual points or understand what's not possible. Then you can actually come up with the right explanations for the design.

alexhans··on OpenClaw: When AI Agents Get Full System Access. Security nightmare?
I'd argue that the moment one cares about accuracy and blast radius, one would natural want to reduce error compounding from a combination of LLM calls (non deterministic) and it's very natural to defer to well tested determinist tools.

Do one thing and do it well building blocks and the LLM acts a translation layer with reasoning and routing capabilities. Doesn't matter if it's one or an orchestrated swarm of agents.

https://alexhans.github.io/posts/series/evals/error-compound...

alexhans··on GPT-5.3-Codex
Isn't the best eval the one you build yourself, for your own use cases and value production?

I encourage people to try. You can even timebox it and come up with some simple things that might look initially insufficient but that discomfort is actually a sign that there's something there. Very similar to moving from not having unit/integration tests for design or regression and starting to have them.

alexhans··on Company as Code
> Has anyone tried this yet? If not, why not?

Yes (in a minimalistic TDD like way?). [1]

I agree that the company as code can be tested without too much risk if you think about as "model your world drawing a circle around what you own".

You can be your own auditor constantly so that you pre-answer questions you would ask in an audit.

I would advise against the always tempting "we are the everything platform" because that rafely scales and makes you a bottleneck. You won't be able to keep up. It's easier to model the tribal knowledge of your own world knowing where the frontiers are and that can even be a forcing function to simplify and reduce externalities.

- [1] https://alexhans.github.io/posts/series/evals/automate-audit...

Edit: make sure whoever works on it is pain point driven and solves things for themselves and then you will build just enough. A world that is more programmatic and can tolerate fuzzy translations (MCPs/LLMs) and where you can test (evals) the level of quality you need can make this cost effective.

alexhans··on Claude Code: connect to a local model when your quota runs out
You're right. Control is the big one and both privacy and cost are only possible because you have control. It's a similar benefit to the one of Linux distros or open source software.

The rest of your points are why I mentioned AI evals and regressions. I share your sentiment. I've pitched it in the past as "We can’t compare what we can’t measure" and "Can I trust this to run on its own?" and how automation requires intent and understanding your risk profile. None of this is new for anyone who has designed software with sufficient impact in the past, of course.

Since you're interested in combating non-determinism, I wonder if you've reached the same conclusion of reducing the spaces where it can occur and compound making the "LLM" parts as minimal as possible between solid deterministic and well-tested building blocks (e.g. https://alexhans.github.io/posts/series/evals/error-compound... ).

alexhans··on Claude Code: connect to a local model when your quota runs out
What are your needs/constraints (hardware constraints definitely a big one)?

The one I mentioned called continue.dev [1] is easy to try out and see if it meets your needs.

Hitting local models with it should be very easy (it calls APIs at a specific port)

[1] - https://github.com/continuedev/continue

alexhans··on Claude Code: connect to a local model when your quota runs out
Useful tip.

From a strategic standpoint of privacy, cost and control, I immediately went for local models, because that allowed to baseline tradeoffs and it also made it easier to understand where vendor lock-in could happen, or not get too narrow in perspective (e.g. llama.cpp/open router depending on local/cloud [1] ).

With the explosion of popularity of CLI tools (claude/continue/codex/kiro/etc) it still makes sense to be able to do the same, even if you can use several strategies to subsidize your cloud costs (being aware of the lack of privacy tradeoffs).

I would absolutely pitch that and evals as one small practice that will have compounding value for any "automation" you want to design in the future, because at some point you'll care about cost, risks, accuracy and regressions.

[1] - https://alexhans.github.io/posts/aider-with-open-router.html

[2] - https://www.reddit.com/r/LocalLLaMA

alexhans··on Hacking Moltbook
There was a post not long ago about a HN user who wanted to both advocate and help people out of this danger:

https://news.ycombinator.com/item?id=46662304

alexhans··on Adventure Game Studio: OSS software for creating adventure games
Hah! Thanks for sharing. Great to see them doing fun stuff. Just watched a quick yatzhee one. I did play a few in the Ben Jordan series and duty and beyond.
alexhans··on Towards a science of scaling agent systems: When and why agent systems work
Agreed.

Even in the case of a single agent, the compounding of errors [1] can easily make your "flow" unacceptable for your use case. The deterministic where possibe/decoupled/well tested approach is key.

With such a fast moving space I'm always wary of adopting optimization techniques that I can't easily prove and pivot from (which means measuring/evals are necessary).

Slowly but surely, abstractions allow us to use others' deep investments in the matter of coordination without losing control (e.g. pyspark worker/driver coordination) and we can invest on friction removal and direct value generation in our domains (e.g. banking/retail/legal payments, etc)

- [1] https://alexhans.github.io/posts/series/evals/error-compound...

alexhans··on Adventure Game Studio: OSS software for creating adventure games
Among other things, this got me dabbling with software development. It had an awesome C++ like scripting language.

It also used to have a button with a text like "make a game" where you would click what you wanted in the form of checkboxes and when you pressed the submit button it would tell you something like, it's not that easy, is it? Wonder how easier it may be now. :P

Shoutout to classic community games like

- Cirque De Zale https://www.adventuregamestudio.co.uk/play/game/377/

- The trilby series (5 days a stranger) https://www.adventuregamestudio.co.uk/play/game/269-5-days-a...

alexhans··on In praise of –dry-run
Agreed. For me a good help, a dry run and a readme with good examples has been the norm for work tools for a while.

It's even more relevant now that you can get the LLMs/CLI agents to use your deterministic CLI tools.

alexhans··on Evaluations for Testing Agentic AI
> Curious what peeple are doing in the real world

It's hard to answer this as the right thing to do might depend a lot on the environment.

My pitch would be that automation is not a new problem and the fundamentals still apply:

- We can’t compare what we can’t measure

- I need to ask myself, can I trust this to run on its own?

- I need to be able to describe what I want

- I need to understand my risk profile

Once you're there, you can use evals-first in a TDD ish style, and keep control throughout your journey, not over-investing, not losing control and allowing people to join in understanding those fundamentals, regardless of whether they're coming from a very solid software engineering background or not.

To be specific, for cross functional teams with > 90% non software engineers, we invested in programmatic first tools (Think claude/codex/continue.dev cli) over "hype" UI solutions because we knew integration would come through MCPs (and then skills), we designed in such a way that the agents/clis/black boxes were decoupled and no particular change/resource limitation would destroy our ability to continue. Focusing a lot on "not having to maintain" (ZeroOps) whatever we do and not having to worry [1] about manually monitoring.

For evals I suggest you explore 2 different tools, such as promptfoo (Technically no programming required for an eval creator, just yaml) and deepevals/pydanticai-evals (python based) and see how you feel about using either based on your context/team composition needs. Once you start, some things will start to click such as the value of bounding non-determinism [2] as much as possible and you might even have proof for them. Other challenges will come later (such as cost, system tests, sandboxing and more).

- [1] https://alexhans.github.io/posts/series/zeroops/no-news-is-g...

- [2] https://alexhans.github.io/posts/series/evals/error-compound...

alexhans··on Management as AI superpower: Thriving in a world of agentic AI
I haven't had the analysis paralysis problem because I've always been quite decent at restructuring environments to avoid bureocracy (which can one of the most dangerous things for a project) but one thing I've observed is that If operations are not ZeroOps then whoever is stuck maintaining systems will suffer by not being able to deliver the "value adding cool features that drive careers".

Since shipping prototypes doesn't actually create value unless they're in some form of production environment to effect change, then either they work and are ZeroOps or they break and someone needs to operate on them and is accountable for them.

This means that at some point, your thesis of

"The dark side of this same coin is when teams try to rely on the AI to write the real code, too, and then blame the AI when something goes wrong" won't really work that way but whoever is accountable will get the blame and the operations.

The same principles for building software that we've always have apply more than ever to AI related things.

Easy to change, reusable, compostable, testable.

Prototypes need to be thrown away. Otherwise they're trace bullets and you don't want to have tech debt in your tracer bullets unless your approach is to throw it to someone else ans make it their problem.

-----

Creating a startup or any code from scratch in a way that you don't actually have to maintain and find out the consequences of your lack of sustainable approaches (tech debt/bad design/excessive cost) is easy. You hide the hardest part. It's easy to do things that in surface look good if you can't see how they will break.

The blog post is interesting but, unless I've missed something, it does gloss over the accountability aspect. If you can delegate accountability you don't worry about evals-first design, you can push harder on dates because you're not working backwards from the actual building and design and its blockers.

Evals (think promtpfoo) for evals-first design will be key for any builder who is accountable for the decisions of their agents (automation).

I need to turn it into a small blog post but the points of the talk https://alexhans.github.io/talks/airflow-summit/toward-a-sha...

- We can’t compare what we can’t measure

- Can I trust this to run on its own?

Are crucial to have a live system that makes critical decisions. If you don't, have this, you're just using the --yolo flag.

alexhans··on Ask HN: How do you find the "why" behind old code decisions?
The fact that you have to dig through things to find it should give you good hints in what is desired for your own code. You want your WHY comments very contextual and findable.

1. Either literally as a comment in the code. Immediately findable

2. As a description in the commit (easy to find with git blame).

3. As a comment in the Code Review/Pull Request system. This works for very stable things but is more brittle than the other ones. Usually exposes the discussion among parties but the outcome of that discussion should probably be in 1. And 2.

Another benefit of keeping comments in the context of where they happen is that you'll actually remember to update them. Otherwise, they're duplication (code smell) and introduce a risk of divergence.

That's why WYSIWYG standalone word docs or similar for design or manually written wikis are so dangerous for code that lives and keeps changing. The thing that keeps teams tidy is often single sources of truth where you can generate/render other things easily.

---

Reading tests or talking to someone is possible but if you had to do that for everything you might waste a lot of time so while it can be extremely beneficial I don't consider it my thought process.

alexhans··on 11% of vibe-coded apps are leaking Supabase keys
A quick reaction is that there's almost 2 different intents that need to be considered here:

- We want to build a business opportunity around auditing.

- We want to reduce the amount of insecure Sugabase apps.

They align somewhat but decisions may vary based on your lens when deciding how much weight to put for watch.

- IANAL, I assume you can or have assesed legal considerations around passive/active/automated scanning of this nature.

- In a direct world you could communitate the fix automatically to the right target for all your finds and track whether they fix the issue (audit periodically at a non spammy/cost inducing frequency)?

- In the general sense I'd try to estimate where I think the error manifests itself to attempt different solutions and find ways to measure those, where am I fixing the problem?

  - LLM generated code: benchmark and evals to measure which popular programmatic LLMs recommend the right approach.

  - Community recommendations: Make your case within the community to modify the appropriate tutorials.
- Is there something in the core tools (I don't really know Supabase) that would make it less likely for a developer following an outdated or malicious tutorial to do the insecure thing?

Security is always a fun problem to think about once you start thinking about it from an economics lens of rational actors with limited knowledge and varying incentives.

alexhans··on Ask HN: What's Your Useful Local LLM Stack?
Unless I'm misremembering, aider with Playwright [1] works with local models, so you can scrape the web.

Depending on your hardware you could do something like:

aider --model "ollama_chat/deepseek-r1:14b" --editor-model "ollama_chat/qwen2.5-coder:14b"

[1] - https://aider.chat/docs/install/optional.html#enable-playwri...

alexhans··on Ask HN: Who is hiring? (May 2025)
Sure. Just sent you an email.
alexhans··on Ask HN: Who is hiring? (May 2025)
Amazon – Economic Decision Science | SDE II | Full Time | ONSITE London, UK

We’re hiring a Software Development Engineer for our team at Amazon: Economic Decision Science (London, UK).

We're working in an extremely interesting space at the Intersection of Economics, Machine Learning, Statistics and Data Engineering, shaping how our stores operate and evolve.

Come help us build our Factory of Economics, where economists can operate independently and in parallel, unconstrained by tech bottlenecks or engineering bandwidth. The goal: No software rewrites, predictable time to production, early integration, and fast iteration.

About the team: - AWS resources at our disposal, with no red tape. We invest to add value. Fast.

- Zero-ops mindset. We build in a way we can focus on delivery, not maintenance.

- No on-call. No phone paging.

- Lean and pragmatic: No stand-ups or sprints; we work towards clear, business-aligned quarterly goals.

- Greenfield projects and a focus on simple solutions for complex domains.

- Tech stack includes Python, Airflow, DBT, Spark, Iceberg, SageMaker, Lambda, and Bedrock – but we always use the right tool for the job.

Role description and application: https://www.amazon.jobs/en/jobs/2968443/sde-ii-economic-deci...

← PreviousPage 4 of 4