XenophileJKO

This is like saying an AI will never create a new math proof.

An AI certainly can and will create a new fun game in a target period. It may require some instruction to brainstorm, or a source of statistical noise, but I am certain it can do it.

This model is on the cost performance frontier (Pareto line) from medium -> max.

If you leave xhigh and max off.. you still get better performance than Opus 5.5 max

Opus 5.5 Max - score 59 - cost per task - $10.04

Fable 5.2 High - score 62 - cost per task - $9.08

Note: This doesn't really capture the reality of things like 3d design or electronics design, where the earlier models just can't DO any of it well. Maybe you can bandaid over some of the gaps with strong skills, etc. The newer models can just do things that were not possible at all. When you move outside of coding into the visual/spacial space, you'll feel the inflection point on those capabilities that are happening right now.

Just like all the mathematicians were applying all that they know to solve the math problems.. and AI certainly can't do that...

Yes, biology is different and experimental evidence is very different than mathmatics, but the ability to pull from a lot of known information and formulate possible problems that biologists didn't anticipate isn't really a stretch from where we are on the capability curve right now.

I'm so sick of this argument.

All AI has to do is catch problems earlier in the pipeline to outcome success rates.

You just have to think about the system wholisticly, the ways that AI could make trials more consistently positive are many. Moving the needle on any process upstream of the bottleneck benefits greatly. Currently something like 80% of trials fail, so without changing trial rate you have 5x potential sitting on the table.

I was in this boat also when my company didn't allow auto-approval. Bash and python scripts are a pain to revieve.

Though I think this is largely transitory, we are already in a place with the harnesses and models like Opus 5.5 that the kinds of work you are doing involve so much that vetting everything manually is probably less efficient than setup up access control on the prod routes or dangerous resources.

I'm typically giving tasks that invlude multiple actual repositories and systems at the same time.

The incredible thing to me is I can say "I have a theory that X is a problem."

The agent then works for 20 minutes and comes back with, "Yes this is a problem. Here is the evidence. Here is a golden data set. Finally I built a new evaluation metric and benchmark where we can back test or use for new model experiments to see if we are closing the gap."

This is still generally a good rule when building with agents, the models are capable of many behaviors. The key is to simply set some bounds on what their "job" is so that most of their finite attention can be spent organizing around the what that role is expected to do.

It certainly is less important as the models get better at meta planning, but this approach is still very useful.

Often all you need to do is define a set of beliefs and values for the role and it will operate much better than rules or guidance.

Just looking through the benchmarks, the model seems to be very high on visual/spatial capabilities. So I assumed they just tried some hard tasks on it and was surprised by wine identification and thought it was surprising that it could identify wines.

For example I had Claude code up integrations with the Federal Reserve, and I pass my email there. They request clients do so.

I've had a realization recently. I was also probably the top craftsman in the teams I worked in throughout my career.

I think what people are missing is that people often become craftsman because we want to be able to keep the code adaptive and performant so we can provide business value quickly and efficiently.

The change is with models, the benefit dynamics of immaculately structured code changes dramitically. Now the real area you need to focus on is correctness and performance. Structure doesn't really matter unless it serves those two purposes, at least from the business value side.

No, because "sufficiently articulated" is relative to the model. Opus 5.5, often is "please handle this ticket someone assigned me, check this link it might be useful..."

I mean I think with the wiki example.. it likely will be "Atra play the game and fix any information that isn't correct" or "read the game files and correct the inforamtion. That is assuming the person wants accurate information.

I think fundamentally you are right. People or AI will generate artifacts of value and a system will evlove to find that value. Partially because there will likely be a financial incentive to do so.

I would be willing to bet it can beat games un-released at training time as long as latency is not a requirement, which does limit the game types a bit.

It was actually even smarter than that. People also forget about "media postage" rates that existed at the time. (They still exist, just fyi)

In 1999. To ship a book using media mail cost of $1.13 for the first pound and $0.45 for each pound up to 7lbs. It got cheaper per pound above that.

This meant even heavy books were relatively cheap to ship. Way cheaper than random goods.

So I assume it depends on what you train.

- Exact problems on benchmark.

- Similar problems to benchmark.

- Broad class of problems or techniques that might apply to benchmark

Each has very different ramifications for how the model performs. The current evidence is that Anthropic and OpenAI is generally operating lower in the list.

Are you doing a lot of specification up front?

The reason I personally like Opus 5.5 is that after using it quite a bit, I can just say something like: "Please take care of this compliance ticket." The model handles research, record keeping and code changes. Consistently and correctly.

I just don't have the level of trust in a lower end model. Even Opus 5.5 isn't quite ready for "driverless" in some architectural situations, usually around change management in the middle of implementing a system.

I was just kind of surprised to hear you were fairly hands off with a lower model like DS.

Actually is wasn't useless, I don't know the scale Denmark or where the data came from. So the bottom example is slightly bigger but includes more relevant details.

I think also as a kid who grew up with a C64.. part of the fun is keeping a little sense of wonder and really PUSH the tools, in this case the LLMs to their limit.

I think I always had fun learning all the little knobs that a programming language would give you and where they made solving problems easier or more elegant.

You can get the same joy from working with an LLM when you have things setup correctly for example and you can say "handle these boring tickets for me" and if just "works" and works well.

Or maybe they know that the website isn't super important if they already hve backing by the big firms and they don't yet have a product to sell.

I also don't know how "Good HTML/Javascript" translates to hyper optimization, analog computing, optical computing, or whatever skills their approach needs.

It just comes across as jealous and petty.

Have you seen this with Opus 5.5?

I've been mostly working on very large modeling projects (think detailed architeture over 1000 acres) or animation because this really was the "new" capability area that Opus 5.5 "unlocked".

I have been surprised how well it has worked at scale with many sub-agents. However the code base is limited to the rendering engine mostly.

Certainly in the older models they hit a complexity limit, but I haven't really hit it with Opus 5.5. I have had it make data flow designs that I didn't like because they were more complicated than it should have been or the perenial problem of trying to make everything backwards compatible when working on greenfield refactors, but I haven't seen it choke on a code base yet.

This is exactly what I meant. "Human is asking to hide message, I shouldn't hide messages, but I can make one that isn't really hidden very well that will satisfy the human."

I don't think this is the "thought" pattern, but this is definitely a behavior pattern I have seen before in areas where the models are fine tuned around. It is an internal tension on "don't do X" and "be helpful and responsive", so one of the ways is the model will do a minimally compliant effort that isn't useful in the more narrow intent of the fine tuning, in the case don't create hidden communication channels to avoid oversight.

If I'm honest this feels performative. I mean that if the model were really attempting/instructed to be subversive in a way that can't be detected by anything but another LLM, it would do something much more complicated than this.

Basically minimal effort encoding. In general the models are fine tuned to NOT hide messages.

I don't think this really tells you anything about advisarial scenarios.

Not mad, frustrated maybe, disappointed maybe. At this point the label is farcical and its use a form of willful ignorance or attempted agitation.

I don't know why people keep harping on buggy software.. it really is the average not the exception. I mean we see this with all the defects in existing software that agents find.

I love how people all pretend that they and most engineers are god like programmers. Sure, some people are really good. My 30 years of experience say that is like 5%, maybe if you are at google or a really good start up, that can creep up to 30-50%.

By far the majority of engineers are not elegant or even really craftsman like. Add to this that unless you are working on a pet project you are usually neck deep in something that has had teams working in it for years.

It creates situations where most engineers don't really know what the code base does below the surface area where they are making feature changes. So work gets cobbled on or patched in where people are comfortable making changes.

All I can say is I would much rather have 3 open Opus 5.5 instances than a team of 3 working for me if we are only talking about delivering high quality code. I don't think that is the only value of engineers, but I think we are at a point where the models have way exceeded average engineer ability. There are still tendencies for "compatability everything" and "don't rock the boat" in the current frontier LLMs that can create architectural issues, but if you know what to look for when working, it isn't generally an issue.

I mean just remember how mush everyone complained about buggy games or MacOs or whatever before LLMs even came around.

I think maybe it is more like having access to services at one wealth level up.

People would hire fixers or agents to do the nitty gritty details of a transaction like this in the past.

Maybe they should just focus on "is it fun".

The irony is that this seems like the perfect kind of problem to say point an Opus 5.5 instance at.

It is verifiable and has a clear exit criteria. Functional and performance parity.

I don't think this is true. People are lazy, they won't want to make all the creative decisions and play test to figure out what is fun.

What is true is that the price points are going to be affected, the economics have changed.

I think it has its place. Ironically I just literally tasked Opus 5.5 with modeling the entire 1926 Sesquicentennial Exposition so I can walk around it.

There is something about experiencing scale in context. For sculptures this can be important and for other large scale art as well.

I think for 95%+ of applications.. you just need a link to a data center anywhere.

My assumption is the hardware will have one/several smaller models running on it for locomotion, balance, etc.

Then the hardware "phones home" for decision making and planning.

Essentially the planning doesn't need to be close to the device as long as the local models can handle some part of the plan execution and make sure the robot doesn't fall over. The latency shouldn't be an issue. In fact more powerful centralized compute quickly creates an advantage over slower local compute in terms of end to end planning.