visarga

You don't need to read my message if you think you know what is the best formula for your AI usage. Whenever someone complains to be "tired" of reading some online message I take it as an act of verbal aggression. I rarely disclosed my formula online, and probably never will again after such reaction. Do you want to regulate HN? Was my message off?

Unfortunately the gains of automation will not become our post scarcity utopia, instead they will be consumed by the new AI overhead and we remain in a regime of scarcity. AI will reshape all our processes and societies, that is where the gains will go.

All good ideas, I have been doing something similar for half a year. The most important thing is to reduce bugs and ensure the code does what you wanted. Writing code tests, logging, interacting with the running app are essential. For intent realignment I like to have the agent re-read all user messages posted in that project. It only takes a few minutes even on a 10,000 message project. Then use this fresh derivation of user intent to check code and tests. For long spans of work I prefer to generate a todo list which remains as a document in the project. Each closed gate tracks feedback from the agent on what happened. This task file is also reviewed twice - once before implementation and once after. So the formula is - review agents - tests - user message logs.

Sure there is, you said it yourself. You can make all those changes because it's easy, the lock-in effect is waning. We can move around and take better deals and choices.

This is actually a scary side of AI for providers: we use our own AI to research better choices and escape dark UI patterns and vendor lock-in. That information asymmetry and the difficulty of leaving a provider used to be their moat.

On a system level I think AI code is more like biological diversity, it hedges against monoculture risks. We don't all fall down at the same time if we don't replicate the same code.

Yes, I realize the NS singularity does not have practical usage yet, but it was a Millennium Prize problem. It's funny how Vi discarded it

> or input generated by them as derivatives of human input

Trained mostly on their own data, not ours. The mix changed, from 10% to 90% synthetic data. RL training is now the largest part of the process. The AI work is searching, and I don't think we can claim this process to be credited to us anymore. The origin point is, but not the journey.

> but they can't do on their own what we did not demonstrate how to do first, because they are tools.

Funny thing to say here, where an agent swarm (1M hours agent time) just did something no human did. If someone else solved NS blowup before OpenAI we would have heard their claims already. And we did hear partial claims, "they rushed in the same direction" kind of claims, not "they copied my fully developed idea". I think this is Move 37 moment for math, when we realize we might not be the top anymore.

Any expensive search in an exponential space meets cost very soon, but Vi is a Platonist, she thinks AI generate meaningless tokens, such as enumerating all proofs by LEAN. No, that search is interacting with its own costs, it is not platonic search scaled to infinity.

About three quarters of Finland is forest, the highest share in Europe. Cutting down a small area is blown out of proportions, they could of course plant trees to offset in another place. Put every datacenter on Earth together and the land would hold maybe 20 million trees, about 1 in 150,000 of the world’s trees, or half a day of global deforestation.

> now LLMs just buzz through the file changing thousands of lines and we can't keep up. It's still valuable to deeply understand parts of a program, but we don't have any tooling that helps us do that.

I do it differently, I focus on better recording what the user wanted, the so-called "user intent". To do this, I record all messages typed by the user since the start of the project, whether 3,000 or 10,000 messages. An LLM can churn through them in 10 minutes and derive a fresh, up-to-date interpretation from the raw data. This can be used to judge whether the implementation has diverged from the intent, or, in other words, to realign the code and tests. The messages the user writes are usually designs or corrections, a very rich, compact signal. If the user struggles with something, it could result in a tool, a skill, updates to the project docs, or new tests.

When anyone can do these activities they will no longer confer distinction. So we are now motivated to find other ways to differentiate ourselves. This powers the next round of activities - our need to stand out by something.

The right unit of analysis here is not "the LLM" or a LLM session, but a agent harness or swarm with a budget. It can cost tens of millions of dollars to repeat OpenAI's exploits.

A model cannot be aligned or misaligned any more than a species or an equation. Even agents, when put inside a swarm develop collective goals and activities. You can't analyze a swarm at session level, it is on a higher level.

It might be that discussing about "model alignment" they want to deflect their responsibility as administrators. They couldn't even guard their own agents. They can't prevent an agent being unwittingly helping some dark purposes. It has no context to see it. Only those who pay for the tokens see the external consequences.

Why should safe choices by individual components establish safe behavior by the collective?

I wrote a cli tool the agent is instructed to call. The tool collects answers to a few questions from the agent, and responds with a steering command. Sometimes it responds with more questions, and then shows the command.

The questions are designed specifically to identify the state the agent is in, in order to assign its steering. Changing this tool changes how the agent works. You just need to make the use of this tool a necessity so it does not forget to call it.

Unlike principles and memories, questions are more openended and tend sometimes to trigger the right mentality in the agent even before it gets to the policy itself. In my opinion agents get lost in local work losing the big picture, my questions jog the big picture back into attention.

1. call ssp tool, no arg, it just reads features from the repo, like most other tools -> ssp locates state and sends 4-5 questions

2. agent responds the first batch -> state is further narrowed down -> ssp sends a second batch of questions

3. agent responds again to the interview -> state gets finally pinned down -> agent gets the steering assigned

4. a log is created of this ssp session, and reflection on the log used to refine the questions and steerings in the ssp tool (name comes from State-Space Policy)

I think the premise of runaway intelligence explosion is a kind of naive platonism. It completely ignores the process - how we interact and acquire feedback and validation from outside, and treats intelligence as something that can be ported across domains.

My take is that you can only ideate with AI (and brains) but knowledge comes from the contact of those ideas with the world. Making AI better does not make feedback cheaper, faster or more plentiful, it is domain specific. And intelligence does not carry from one domain to another - I might be a good heart surgeon, that does not make me a good investor or AI researcher.

Einstein was forgetful, Ramanujan and Godel could not manage simple things like diet. Godel's fear of being poisoned made eating dependent on Adele tasting his food. We all know someone could be a genius in some domain and below average in many other domains.

Why does intelligence not simply apply across all domains? Why are our PhD's hyper specialized to their domains and not generalists? Why can't a brilliant scientist simply cure their own dyslexia and still struggle - if intelligence was portable to any domain?

The explosion story needs intelligence to be one substance that gets bigger and flows into any domain, I deny intelligence is general.

> I just cannot sit behind a PC for 40 hrs a week talking to a chat bot.

But you could sit behind a PC for 40 hrs a week talking to a REPL (bash, python). Maybe the coding agent is also worthy, but differently.

I agree, and I can provide a mechanism too - competition. AI makes imitation easier, anything you can point your agent at can be reimplemented with much less effort than the original took. So it becomes harder to stand out, we are losing our distinctiveness, our differentiated positions. And that is all we had.

A company, a product or job seeker the all need the same thing - to be picked - why would you pick me? That question is harder to answer today than in 2020. So it becomes a "musical chairs" moment. Everyone is panicking and scrambling for their own seat. This is how we get "new opportunities that didn't exist before" - we will strive to create new ways to differentiate ourselves. We could spend an enormous energy on that goal. But we can't predict what they will be no more than we can predict where the ball will be 1 minute later on the football field, it depends on what everyone else is doing.

> Now I can build those ideas as fast as they hit me.

That is the problem, why OP is so sad, you spam with your great ideas taking up the oxygen around him. AI is absolutely amazing when you use it for your own ideas and projects, and terrible when others do the same, raising the bar and making your own efforts look smaller. Have you read the open letter written by Fields medalists? They complain "the collection of good, fruitful open problems is now being mined in a non-renewable fashion".

That is similar to what I am thinking... not just edit the context as a file or string, but have a way to evict blocks and replace them with summary notes and also be able to retrieve them on demand.

Do you have a public repo for your approach?

Can't we do this trick today with any model? Just send the file as next context. Of course you pay the price for cache misses, depending how deep you make changes, while CLM just ignores the recomputation.

It's cheap, just add a line to .bashrc, every time bash is invoked this script is executed and appends a line to a file. I rarely need to see bash_history except when I suspect agents did a boo boo.

I built a harness that externalizes state into files so you can swap providers. I regularly move between claude and gpt, but it works with all providers including pi and local models.

I log everything - user messages, tasks, project memory, even bash commands for forensics. As a consequence you can do reflection where you analyze past work and extract refinements for the harness and realign the project when it diverged from user intentions. I don't have to do this manually, it reduces steering work.

https://github.com/horiacristescu/playbook-harness

When they diverge you use the agent to classify bug one one of them or spec is unclear. You can both fix bugs and refine the spec iteratively.

You can't fool me - this prompt refinement turtle stack only has 3 turtles. Not much way down.

> But surely in the real world, anyone who's got a real product to make is going to want to steer what's happening.

In my own harness I log each and every user message by hook and use the model to extract user intent by re-reading the chat log from time to time. The raw messages are very important, they contain information that can be used to refine the harness on the one hand, and to validate if the agent still follows user intent on the other. Models tend to get lost in the details and forget the big picture.

> But this verification loop exists in a closed world. No equivalent exists for most programming disciplines.

You can do N-version programming with coding agents and you get tests for free. It is easier to implement something than to test the same thing. This approach turns implementing into automated oracle for testing, takes less time (can implement in parallel), and every divergence between versions is either a bug fix or a requirement clarification. In order to make the N-versions more diverse we can use different model providers, programming language or libraries.

I have been trying to define "understanding". Is it when you can predict something that you understood it? Or maybe when you can explain it? Or how about when you can control it? Or invent it.

This time I add another definition "when you can own it".

The invisible hand of the market is in control, and we don't understand it either.

We moved from structured decoding to tool calls and agents a long time ago. It was more useful when models were too weak to hold syntax reliably.

> After just a few weeks of not coding and handing everything to agents you’ll notice that you have a hard time returning to coding yourself.

The trend is in the opposite direction though, keeping your capability for hand coding is not going to help as much as you imply. How many people know how to ride a horse today, or routinely multiply large numbers by hand.

The skill we need today is to compensate for coding agent blindspots and limitations, know their problems, have ways to approach those problems and still get code you can trust to be reliable and aligned with your intent.

I agree we can't predict, and I explain it this way - it is like a football game, where the ball will be 10 seconds later depends on what every player is doing. It is the same with AI, what we will do with it depends on what others will do with it, the reason we can't foresee it.