enraged_camel

No, not really. Yesterday I asked Opus if it can read the logs from the sessions I have on the local ChatGPT app. It looked around and said no, that’s not possible. I said “what about these jsonl files in this folder?” It read them and said “ah yes these seem to be it!”

So I went ahead and created a skill for it. This is so that future Opus agents won’t come to the wrong conclusion the first one did. I can say “read the codex session titled ‘X’” And they will know exactly what to do and do it effortlessly.

I’ve used Astra for the past day and a half. My layperson’s review is that it is impressive at computer use and 3D reasoning, and fails in similar ways to 5.6 Sol at similar rates when it comes to coding. I have no idea how it scored so high on SWE benchmarks because so far it has been very “mid” as the kids say.

You are falling for selection bias.

For every person using Astra to create a game from some random idea and sharing the impressive result, there are an unknown number who have tried the same thing and gave up in frustration.

There is something deeply wrong with Astra. I can’t quite put my finger on it. On the one hand it is a lot more knowledgeable, which makes sense since it’s a larger model. On the other hand that knowledge doesn’t reliably translate to intelligence or insight. Certainly tasks like 3D modeling it does extremely well. Other stuff like complex coding problems in an existing codebase it stumbles more often than not. This morning it ran around in circles. It implemented a feature, then convinced itself that it should have followed “proper TDD”, deleted all the code it had written and wrote 8,500 LoC of unit tests. At that point I was down to 35% of quota so I stopped it and gave the task to Opus 5.

Really weird model. No idea how it did so well on all the benchmarks.

Yeah their support is non-existent. It's actually mind-blowing that so many people use it.

This whole thing is an absolute disaster honestly, and yes it is being downplayed and hand-waved away.

Since March, so many people have mocked Anthropic for their approach to Mythos release, claimed it was all marketing, accused them of holding back the best models from the general public to boost their revenues and upcoming IPO, etcetera. Yet these OpenAI revelations offer a small glimpse into the type of world we would be in if everyone had full access to these models from day one.

OpenAI was desperate to catch up, and no doubt under tremendous pressure to do so. That's why they were so reckless with their training. They have been doing damage control and reputation management, talking about how important alignment is and how they will slow things down and so on, and have seen the light in terms of holding back cyber capabilities from everyone except a select few. So in a sense, Anthropic has been fully vindicated.

I wonder if OpenAI boosters (and employees) will ever admit this and publicly apologize.

>>> The demos of Fable/GPT-6 are impressive, but "real AGI" should act more like a collaborator than either a peon or overachiever.

I don't really agree. The thing that makes Fable feel like an actual collaborator is its ability to sus out your real intent when you give ambiguous instructions. It's really good at it.

I watched some reviews today and came way with the impression that Astra is not better than Sol in this regard. You still have to be very specific with your instructions. For example, you can say "why is it not committed yet?" and it will give you an explanation and say it's actually ready to be committed. But it won't commit unless you explicitly say so.

That sounds like a very tedious way of working with AI agents, but I understand some people want a high level of control.

>> I'm genuinely so confused when people say this with a straight face. Are you talking about coding? Desktop use? Prose? Or something else?

Same. It makes me wonder what types of things the person must be working on.

I can't speak for others but I have a feeling you're in the very small minority with this take.

You could say Sol is faster and cheaper and that's true. Outperforms Fable? Impossible to believe without hard evidence.

They are desperately, desperately trying to make a name for themselves as the lab that first created AGI, because Anthropic's IPO is just around the corner.

They used a custom harness. It's not a one-to-one comparison.

Yep. Incredibly misleading. Although it is not surprising at this point. They are desperate and will do anything to undermine Anthropic's upcoming IPO.

Exact same thing happed to me. I gave it a small/medium-sized ticket, walked away, came back to a 25,000 LoC monstrosity that both Fable and another 5.6 Sol agent said is 98% useless and should be thrown away.

Paywalled.

>> The way they keep spinning off these niche offerings tells you one of 2 things, it's either working pretty well, or it is failing very, very badly.

You posited two extremes, with no evidence for either, and with no other possibilities in between, or even outside the spectrum. That's beyond mere conjecture: it's just nonsense. I hope you realize that.

>> Perhaps switching costs are greater than some would believe.

I haven't switched because there's nothing to switch to that is anywhere as good. I've been making dedicated attempts at using Sol but it falls short, despite what some people claim.

There are so many deep flaws with this article that it is difficult to believe it was written by a seasoned economics professor. I can only conclude that he's pushing some sort of agenda or is otherwise politically motivated.

1. The complaints he is mocking ("can't afford a $1,000 repair", "can't afford eggs") are about liquidity and cash flow. He rebuts them with accumulated net worth. Home equity or 401k can't pay for groceries though. Someone can both have decent net worth and have trouble affording basic necessities or paying for emergencies.

2. Households aren't people. The SCF's unit is the family, and the age is set by the reference person. The author says stuff like "Americans at the 75th percentile are millionaires by their mid-50s" describes households that usually contain two adults. Per person, divide that number by two and it becomes a lot less impressive. It also mangles the young end, e.g. a 25-year-old living with their parents aren't their own household.

3. A cross-section isn't a life cycle. He reads a 2022 snapshot as a trajectory ("age a couple decades and you're rarely poor"). But today's 65-69s are special: they caught a four-decade bond bull market, an equity boom, and a housing boom on top of that. Add survivorship: poorer, sicker people die earlier and drop out of the older bins, which mechanically inflates old-age wealth.

4. He concedes half of median wealth is home equity, then waves it away. But if you strip primary residence, the overall SCF median falls from about $192,900 to roughly $57,900. Reverse mortgages ("negative mortgages") have steep fees, low loan-to-value in your 60s, and tiny take-up... and you still have to live somewhere.

Aside from all that, he is also using the data wrong. He had ChatGPT build the table from the raw SCF. I am actually quite familiar with that file. It requires survey weights and five imputation implicates handled correctly. Percentiles in five-year age groups from ~4,600 families carry error bars that are pretty wide! Not to mention 2022 was a peak: median net worth had jumped 37% from 2019, which was the largest on record, and it happened mostly due to house prices skyrocketing. That, in turn, skewed the numbers.

I disagree. As the saying goes: never attribute to malice what can be adequately explained by incompetence. That goes for both OpenAI and HF (but mostly the former, as the latter was the victim).

You are either seriously naive, or have never worked on any large and complex legacy codebase.

Yes they are.

The OP doesn't appear to know what they are talking about. Fable can absolutely be used to develop applications. It's just that for security stuff I use Opus 5. Which is fine for most use cases.

The thing that blows my mind about it is that they straight up copied Claude Code and tried to frame it as some sort of incredible accomplishment. There are no new ideas, no insightful UI/UX paradigms, no killer features. They say money can't buy taste. The ChatGPT/Codex app is proof that it cannot buy creativity either.

From the article:

"We plan to make Astra available soon, but access to its most advanced cybersecurity capabilities will be more limited. Advanced cybersecurity work will initially be available to a group of testers, with access through Daybreak Blue following to expand defensive use."

This, after several months of OpenAI and its boosters relentlessly criticizing Anthropic for withholding Mythos from the general public, is laughable.

Sam, just three weeks ago, posted this tweet: https://x.com/sama/status/2085862292311396515

In the tweet, he said: "we do not think it is a good strategy to keep powerful models to a chosen few."

And yet here we are.

I wonder if he will demonstrate good character and admit he was wrong.

What predictions have Ed made that were correct? And how do they weigh against the incorrect ones, in terms of both quantity and quality?

That's relevant because if you make a thousand predictions, a few of them might turn out to be true. That doesn't mean you're good at making accurate predictions.

If you find derision and extreme cynicism "soothing", the problem may be with you, rather than those who are optimistic about AI.

>> I think it's fair to call his specific predictions "early." I'm saying his timelines are too short.

That means his predictions were either wrong, or meaningless.

In this case, it is. Zitron does not possess even the most basic knowledge about the issues and practices he complains about. For example: https://www.crisesnotes.com/sigh-no-ed-zitron-ai-bond-issuan...

In a way, this is the only benchmark I care about now. :)

Great, thanks for sharing.

>> Fable easily trips its safe guards.

Maybe it depends on the type of work you do, because for me it almost never happens.

>> You can be 95% complete with the plan for it to trip and then lose it all.

That's... not what happens though. The session will either seamlessly downgrade to another model mid-session, or it will stop with an alert and you can just re-prompt it. It will still have access to the context.

Interesting that they seem to have gone all-in on science, and life sciences in particular. Improvements to coding performance seem marginal, although cost savings are very welcome.

Curious to see how Astra does.