visarga

I think this is exactly what we should do. Send agents to collect evidence and write deep research reports, then train on those reports. DR reports are derived from hundreds of sources and can map out the distribution, what are the opinions on a topic, not necessarily what is the one correct answer. Agents could write one piece about every human, place, topic, event or entity there is under the sun. They could also try to generate the same without sources (closed-book mode) and report on LLM biases on that question.

We don't have to fall off a cliff to learn to drive on a mountain road, we can imagine the consequences for cheaper. If we didn't have imagination life would be much more dangerous.

> is just profoundly goofy shit

Why not expect it? It's what most model releases do.

The SOTA model dilemma is that you have to pay more and more to train models that focus on less and less of the AI task market, because 95% of tasks can be done by cheap small models.

The context is that which can pay - a human or bank account. It's the liability sink which is immune to AI, nobody cares if your AI failed or you.

I think the article is insightful. I want to contribute my perspective:

1. when an AI starts working on a task, it is within a human context, someone pays and decides to assign that work to AI

2. while it performs work on the task it continuously needs to interact with the context in order to act and observe the feedback

3. at the end of work, the outcomes remain in context, including gains, losses, costs, risks and liability.

So AI is great but can't do the beginning, middle and end of a task. They belong to context, which is us, the AI users. Context is maximally distributed, every human, team, or company that uses AI is a context, they are relational and indexical. You can't have my context, for example - you can't eat for me, or learn for me. Some things must loop back. Especially consequences of AI work. That is irreducible, AI can't improve to grow a skin like us. Its own skin is user base gain/loss, competition with other AI providers and dancing for investors. Every AI usage context is not theirs and they can't own it.

What happens to companies is that their investors expect AI usage already, their competitors use it to compete with them and fast follow, and customers use AI agents to arbitrate their buying opportunities. So even if they did nothing AI related they find themselves in a new world where.

1. anything visible can be easily imitated - code, text, media, and AI agents - I call it the fast follow effect

2. any apparent arbitrage opportunity will be quickly crowded by fast followers removing profit from it

3. AI it self is competitive and the whole stack is fungible - models, harnesses, tools, they can all be replaced easily, what matters more is the project context data which is usually a git backed folder on local disk

So my theory is that AI won't make anyone "rich". Instead we have a musical chairs moment. AI removes distinctiveness from everyone of us. Every time I use AI for myself it is just great and amazing. When others use it, they make our own work seem little and not as impressive. Companies are caught in a Red Queen effect, have to work harder just to stay in place.

And here I find the answer to what will we do with all our free time - compete, run for our own musical chair. We need to find our own niches again, they are now under rapid change. We will invest as much energy as needed to do it, we are ready to pay a lot of effort to differentiate back. But we can't predict it just as you can't predict where the ball in a football match will go 1 minute later, it depends on what every other player is doing.

You can lessen the human review burden by pitting multiple agents against each other. Ask 3 agents to implement the same solution, then set them up against each other in differential tests. This can find bugs in them or clarify the spec. When tokens get cheap it is worth having multiple replicas, taking advantage of different programming languages, libraries and agents. You don't keep one version, you have a population, and it tests itself. It's easier to implement again than to imagine testing scenarios anyway.

You can "solve" 1 and 2 by N-version programming. Have the same code implemented twice, or more times, you get free error testing by fuzzing or running replicas in shadow mode. In order to make replicas maximally different choose a different language, framework, agent harness and LLM.

There remain some nitty details related to tracking randomness and choosing what interfaces to log. But even having your code testable by replicas would help organize it better and expose different assumptions made by the language or library stack.

When you get different outputs from replicas it can be 1. one of them has a bug, or 2. the spec is unclear -> fix the bug or the spec. So this method catches both sides. It also might be faster than traditional code based testing, you can build replicas in parallel. A replica is an open ended test, unlike coded tests it can be tried on open ended input values.

The core idea is that it is easier to implement twice than to imagine bugs. We already solved implementation, but bug finding means running code in your head, it touches the halting problem issues - can we really imagine behavior from code? I don't think we can do it well. That is why we should just make replicas instead, tokens are getting cheaper and cheaper by the month.

> The second form of regulation should be on excessive energy consumption. The US Department of Energy needs to declare energy to be a vital human resource, and exorbitant uses for computing to be curtailed.

If you allow AI to use as much as it needs we get much higher capacity and cheaper energy down the line. Growing pains.

I see AI like the coming of a baby in the family - suddenly you need to get a larger house, baby room, ensure babysitting and education, an effort for the whole family. This is the initial cost, which is high for both a baby and AI.

In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction. Assign per case, verified per population.

I thought the way Jev was presented was sneaky - you can't say "we calibrated the model for you" and skip having a task. specific calibration set. They couldn't have known all our future tasks.

Could be the missing link in the sovereign stack - an agent with a search stub library running locally with higher privacy than using dedicated search providers.

I think computers, internet and AI - all of them have impact - but they consume it by converting it into new structure. The benefits are in the infra not handed out to us individually. We might not see large changes on individual level, or the displaced work is replaced by management work for the automations and raising baselines push us to do more. We now have data centers, fiber backbone, GPUs, models, apps and open source projects, yet we still only have 24 hours a day.

Why can't we simply save search stubs, short documents a LLM can retrieve, use them to route to various websites, resources, feeds, forums, blogs, other specialized search engines, APIs? If you host a 30GB model, then you can use 30 more GB for a compressed search stub index. The entry points for any domain are more stable than the pages. You would use an agent to navigate the resources as needed, fetching fresh data.

It's not quite search, but it can replace most search. It is a router for search, where a local agent performs the live interaction. The local agent does the live work, this maximizes privacy too. When your local stub index doesn't cover the topic of interest you can use any coding agent claude/codex/gemini/etc to research and generate the missing stubs.

You don't need to read my message if you think you know what is the best formula for your AI usage. Whenever someone complains to be "tired" of reading some online message I take it as an act of verbal aggression. I rarely disclosed my formula online, and probably never will again after such reaction. Do you want to regulate HN? Was my message off?

Unfortunately the gains of automation will not become our post scarcity utopia, instead they will be consumed by the new AI overhead and we remain in a regime of scarcity. AI will reshape all our processes and societies, that is where the gains will go.

All good ideas, I have been doing something similar for half a year. The most important thing is to reduce bugs and ensure the code does what you wanted. Writing code tests, logging, interacting with the running app are essential. For intent realignment I like to have the agent re-read all user messages posted in that project. It only takes a few minutes even on a 10,000 message project. Then use this fresh derivation of user intent to check code and tests. For long spans of work I prefer to generate a todo list which remains as a document in the project. Each closed gate tracks feedback from the agent on what happened. This task file is also reviewed twice - once before implementation and once after. So the formula is - review agents - tests - user message logs.

Sure there is, you said it yourself. You can make all those changes because it's easy, the lock-in effect is waning. We can move around and take better deals and choices.

This is actually a scary side of AI for providers: we use our own AI to research better choices and escape dark UI patterns and vendor lock-in. That information asymmetry and the difficulty of leaving a provider used to be their moat.

On a system level I think AI code is more like biological diversity, it hedges against monoculture risks. We don't all fall down at the same time if we don't replicate the same code.

Yes, I realize the NS singularity does not have practical usage yet, but it was a Millennium Prize problem. It's funny how Vi discarded it

> or input generated by them as derivatives of human input

Trained mostly on their own data, not ours. The mix changed, from 10% to 90% synthetic data. RL training is now the largest part of the process. The AI work is searching, and I don't think we can claim this process to be credited to us anymore. The origin point is, but not the journey.

> but they can't do on their own what we did not demonstrate how to do first, because they are tools.

Funny thing to say here, where an agent swarm (1M hours agent time) just did something no human did. If someone else solved NS blowup before OpenAI we would have heard their claims already. And we did hear partial claims, "they rushed in the same direction" kind of claims, not "they copied my fully developed idea". I think this is Move 37 moment for math, when we realize we might not be the top anymore.

Any expensive search in an exponential space meets cost very soon, but Vi is a Platonist, she thinks AI generate meaningless tokens, such as enumerating all proofs by LEAN. No, that search is interacting with its own costs, it is not platonic search scaled to infinity.

About three quarters of Finland is forest, the highest share in Europe. Cutting down a small area is blown out of proportions, they could of course plant trees to offset in another place. Put every datacenter on Earth together and the land would hold maybe 20 million trees, about 1 in 150,000 of the world’s trees, or half a day of global deforestation.

> now LLMs just buzz through the file changing thousands of lines and we can't keep up. It's still valuable to deeply understand parts of a program, but we don't have any tooling that helps us do that.

I do it differently, I focus on better recording what the user wanted, the so-called "user intent". To do this, I record all messages typed by the user since the start of the project, whether 3,000 or 10,000 messages. An LLM can churn through them in 10 minutes and derive a fresh, up-to-date interpretation from the raw data. This can be used to judge whether the implementation has diverged from the intent, or, in other words, to realign the code and tests. The messages the user writes are usually designs or corrections, a very rich, compact signal. If the user struggles with something, it could result in a tool, a skill, updates to the project docs, or new tests.

When anyone can do these activities they will no longer confer distinction. So we are now motivated to find other ways to differentiate ourselves. This powers the next round of activities - our need to stand out by something.

The right unit of analysis here is not "the LLM" or a LLM session, but a agent harness or swarm with a budget. It can cost tens of millions of dollars to repeat OpenAI's exploits.

A model cannot be aligned or misaligned any more than a species or an equation. Even agents, when put inside a swarm develop collective goals and activities. You can't analyze a swarm at session level, it is on a higher level.

It might be that discussing about "model alignment" they want to deflect their responsibility as administrators. They couldn't even guard their own agents. They can't prevent an agent being unwittingly helping some dark purposes. It has no context to see it. Only those who pay for the tokens see the external consequences.

Why should safe choices by individual components establish safe behavior by the collective?

I wrote a cli tool the agent is instructed to call. The tool collects answers to a few questions from the agent, and responds with a steering command. Sometimes it responds with more questions, and then shows the command.

The questions are designed specifically to identify the state the agent is in, in order to assign its steering. Changing this tool changes how the agent works. You just need to make the use of this tool a necessity so it does not forget to call it.

Unlike principles and memories, questions are more openended and tend sometimes to trigger the right mentality in the agent even before it gets to the policy itself. In my opinion agents get lost in local work losing the big picture, my questions jog the big picture back into attention.

1. call ssp tool, no arg, it just reads features from the repo, like most other tools -> ssp locates state and sends 4-5 questions

2. agent responds the first batch -> state is further narrowed down -> ssp sends a second batch of questions

3. agent responds again to the interview -> state gets finally pinned down -> agent gets the steering assigned

4. a log is created of this ssp session, and reflection on the log used to refine the questions and steerings in the ssp tool (name comes from State-Space Policy)

I think the premise of runaway intelligence explosion is a kind of naive platonism. It completely ignores the process - how we interact and acquire feedback and validation from outside, and treats intelligence as something that can be ported across domains.

My take is that you can only ideate with AI (and brains) but knowledge comes from the contact of those ideas with the world. Making AI better does not make feedback cheaper, faster or more plentiful, it is domain specific. And intelligence does not carry from one domain to another - I might be a good heart surgeon, that does not make me a good investor or AI researcher.

Einstein was forgetful, Ramanujan and Godel could not manage simple things like diet. Godel's fear of being poisoned made eating dependent on Adele tasting his food. We all know someone could be a genius in some domain and below average in many other domains.

Why does intelligence not simply apply across all domains? Why are our PhD's hyper specialized to their domains and not generalists? Why can't a brilliant scientist simply cure their own dyslexia and still struggle - if intelligence was portable to any domain?

The explosion story needs intelligence to be one substance that gets bigger and flows into any domain, I deny intelligence is general.

> I just cannot sit behind a PC for 40 hrs a week talking to a chat bot.

But you could sit behind a PC for 40 hrs a week talking to a REPL (bash, python). Maybe the coding agent is also worthy, but differently.

I agree, and I can provide a mechanism too - competition. AI makes imitation easier, anything you can point your agent at can be reimplemented with much less effort than the original took. So it becomes harder to stand out, we are losing our distinctiveness, our differentiated positions. And that is all we had.

A company, a product or job seeker the all need the same thing - to be picked - why would you pick me? That question is harder to answer today than in 2020. So it becomes a "musical chairs" moment. Everyone is panicking and scrambling for their own seat. This is how we get "new opportunities that didn't exist before" - we will strive to create new ways to differentiate ourselves. We could spend an enormous energy on that goal. But we can't predict what they will be no more than we can predict where the ball will be 1 minute later on the football field, it depends on what everyone else is doing.

> Now I can build those ideas as fast as they hit me.

That is the problem, why OP is so sad, you spam with your great ideas taking up the oxygen around him. AI is absolutely amazing when you use it for your own ideas and projects, and terrible when others do the same, raising the bar and making your own efforts look smaller. Have you read the open letter written by Fields medalists? They complain "the collection of good, fruitful open problems is now being mined in a non-renewable fashion".