Bjorkbat

Can't take this seriously knowing that this is the same Mustafa Suleyman who...

- Was basically acqui-hired by Microsoft from Pi AI (seems a little biased to recommend a book from one of your own)

- Left DeepMind due to allegations of bullying (https://en.wikipedia.org/wiki/Mustafa_Suleyman#DeepMind_and_...)

- Allegedly yelled at OpenAI employees because they weren't sharing technologies frequently enough (https://www.nytimes.com/2024/10/17/technology/microsoft-open...)

But what do I know, maybe if I read it and regurgitate its contents in a not-too-obvious way I can get an AI policy job.

Tangentially related, I get strong Metaverse/NFT vibes around predictions on agents.

Namely, a lot of predictions were made around NFTs that just didn’t make sense or were kind of dumb. My pet favorite was this notion that in the future you could bring your NFTs with you to different games and the like. You could buy a Batman NFT costume and have your guy wear it while playing metaverse World of Warcraft. They basically took Ready Player One and ran with it. Besides the fact that this is much harder to do than they could imagine, it’s also kind of a goofy idea.

I feel the same way with predictions made around AI agents. My pet favorite is the notion that we stop using the internet and delegate everything to our agents. Planning a trip? Let an AI agent handle things for you. Shopping? Likewise, let an agent handle your purchases. In the future ads won’t even be targeted at people, they’ll target agents instead, and pretty soon agents won’t even browse the internet but talk to other agents instead.

Is it feasible? I can’t say. I’m more interested in how goofy it all sounds. The notion that you no longer have buyer preferences while your agent gets served ads, or the notion of planning that trip to Rome or whatever and just entrusting the agent with the itinerary as if it won’t come up with unoriginal suggestions.

Work agents make more sense in general, but the sentiment remains.

If I recall correctly the authors of the benchmark did mention on Twitter that for certain issues models will submit an answer that technically passes the test but is kind of questionable, so yeah, good point.

I think it's still an interesting way to measure general intellience, it's just that o3 has demonstrated that you can actually achieve human performance on it by training it on the public training set and giving it ridiculous amounts of compute, which I imagine equates to ludicrously long chains-of-thought, and if I understand correctly more than one chain-of-thought per task (they mention sample sizes in the blog post, with o3-low using 6 and o3-high using 1024. Not sure if these are chains-of-thought per task or what).

Once you look at it that way it the approach really doesn't look like intelligence that's able to generalize to novel domains. It doesn't pass the sniff test. It looks a lot more like brute-forcing.

Which is probably why, in order to actually qualify for the leaderboard, they stipulate that you can't use more than $10k more of compute. Otherwise, it just sounds like brute-forcing.

It's easy to miss, but if you look closely at the first sentence of the announcement they mention that they used a version of o3 trained on a public dataset of ARC-AGI, so technically it doesn't belong on this list.

To me it's very frustrating because such little caveats make benchmarks less reliable. Implicitly, benchmarks are no different from tests in that someone/something who scores high on a benchmark/test should be able to generalize that knowledge out into the real world.

While that is true with humans taking tests, it's not really true with AIs evaluating on benchmarks.

SWE-bench is a great example. Claude Sonnet can get something like a 50% on verified, whereas I think I might be able to score a 20-25%? So, Claude is a better programmer than me.

Except that isn't really true. Claude can still make a lot of clumsy mistakes. I wouldn't even say these are junior engineer mistakes. I've used it for creative programming tasks and have found one example where it tried to use a library written for d3js for a p5js programming example. The confusion is kind of understandable, but it's also a really dumb mistake.

Some very simple explanations, the models were probably overfitted to a degree on Python given its popularity in AI/ML work, and SWE-bench is all Python. Also, the underlying Github issues are quite old, so they probably contaminated the training data and the models have simply memorized the answers.

Or maybe benchmarks are just bad at measuring intelligence in general.

Regardless, every time a model beats a benchmark I'm annoyed by the fact that I have no clue whatsoever how much this actually translates into real world performance. Did OpenAI/Anthropic/Google actually create something that will automate wide swathes of the software engineering profession? Or did they create the world's most knowledgeable junior engineer?

> Also, 1 odd thing I noticed is that the graph in their blog post shows the top 2 scores as “tuned”

Something I missed until I scrolled back to the top and reread the page was this

> OpenAI's new o3 system - trained on the ARC-AGI-1 Public Training set

So yeah, the results were specifically from a version of o3 trained on the public training set

Which on the one hand I think is a completely fair thing to do. It's reasonable that you should teach your AI the rules of the game, so to speak. There really aren't any spoken rules though, just pattern observation. Thus, if you want to teach the AI how to play the game, you must train it.

On the other hand though, I don't think the o1 models nor Claude were trained on the dataset, in which case it isn't a completely fair competition. If I had to guess, you could probably get 60% on o1 if you trained it on the public dataset as well.

It's not so much the cost as much the fact that they got a slightly better result by throwing 172x more compute per/task. The fact that it may have cost somewhere north of $1 million simply helps to give a better idea of how absurd the approach is.

It feels a lot less like the breakthrough when the solution looks so much like simply brute-forcing.

But you might be right, who cares? Does it really matter how crude the solution is if we can achieve true AGI and bring the cost down by increasing the efficiency of compute?

I was impressed until I read the caveat about the high-compute version using 172x more compute.

Assuming for a moment that the cost per task has a linear relationship with compute, then it costs a little more than $1 million to get that score on the public eval.

The results are cool, but man, this sounds like such a busted approach.

That’s pretty common and not actually limited to H1-Bs

I know universities will do this with certain open positions where they already have a candidate in mind but are required to advertise an opening, can’t remember the specifics why though. Same with RFPs.

Really easy system to defeat.

I always assumed firing one of these things was a suicide mission, but some NATO guy on Quora has suggested otherwise, and it seems pretty convincing.

https://www.quora.com/Would-the-Davy-Crockett-tactical-nucle...

Basically, even though the W54 had a max yield of a 1kt, the Davy Crockett used a warhead with a far lower yield, as low as 10 tons

Like I said, 3D printers have gotten incrementally better. Buddy of mine makes high-quality prints on one that’s way cheaper than what I owned back in the day.

And yet nothing has really changed because he’s still using it to print dumb tchotchkes like every other hobbyist 10 years ago.

I can foresee them getting better but never getting good enough to where they actually fundamentally change society or live up to past promises

> I am constantly surprised how prevalent this attitude is. ChatGPT was only just released in 2022. Is there some expectation that these things won't improve?

I mean, in a way, yeah.

Last 10 years were basically one hype-cycle after another filled with lofty predictions that never quite panned out. Besides the fact that many of these predictions kind of fell short, there's also the perception that progress on these various things kind of ground to a halt once the interest faded.

3D printers are interesting. Sure, they have gotten incrementally better after the hype cycle died out, but otherwise their place in society hasn't changed, nor will it likely ever change. It has its utility for prototyping and as a fun hobbyist machine for making plastic toys, but otherwise I remember people saying that we'd be able to just 3D print whatever we needed rather than relying on factories.

Same story with VR. We've made a lot of progress since the first Oculus came out, but otherwise their role in society hasn't changed much since then. The latest VR headsets are still as useless and still as bad for gaming. The metaverse will probably never happen.

With AI, I don't want to be overly dismissive, but at the same time there's a growing consensus that pre-training scaling laws are plateauing, and AI "reasoning" approaches always seemed kind of goofy to me. I wouldn't be surprised if generative AI reaches a kind of equilibrium where it incrementally improves but improves in a way where it gets continuously better at being a junior developer but never quite matures beyond that. The world's smartest beginner if you will.

Which is still pretty significant mind you, it's just that I'm not sure how much this significance will be felt. It's not like one's skillset needs to adjust that much in order to use Cursor or Claude, especially as they get better over time. Even if it made developers 50% more productive, I feel like the impact of this will be balanced-out to a degree by declining interest in programming as a career (feel like coding bootcamp hype has been dead for a while now), a lack of enough young people to replace those that are aging out, the fact that a significant number of developers are, frankly, bad at their job and gave up trying to learn new things a long time ago, etc etc.

I think it really only matters in the end if we actually manage to achieve AGI, once that happens though it'll probably be the end of work and the economy as we know it, so who cares?

I think the other thing to keep in mind is that the history of programming is filled with attempts to basically replace programmers. Prior to generative AI, I remember a lot of noise over low-code / no-code tools, but they were just the latest chapter in the evolution of low-code / no-code. Kind of surprised that even now in Anno Domini 2024 one can make a living developing small-business websites due to the limitations of the latest batch of website builders.

I'm weirdly not too surprised due to this belief I have that software developers would make effective criminals. A lot of this boils down to a belief I have that not getting caught in the first place is easy. Murders have something like a 50% solve rate and you can decrease your chances of getting caught with a little knowledge on how to evade common forensic techniques along with some planning. Those who get caught doing one crime or another either were dumb to begin or eventually got lazy and made a dumb mistake in hindsight.

Besides that though, the ethos that we have lends itself well to acquiring advanced knowledge in more-or-less all domains, crime and forensics included.

A tangent perhaps, but I've felt this with AI, albeit with nuanced differences. In a nutshell, there's this weird tension between being kind of dismissive of it due to failed personal expectations, but also a desire to be as objective about it as possible, a "fear of missing out" if you will. As such I consume news about AI and engage with the "community" online about it, but I struggle constantly between separating noise from signal, "truth" and bullshit. It's not so much that bad news gets me down so much as the fact that sensationalist hype and bullshit makes me irrationally upset.

The whole OpenAI strawberry thing is kind of a great example. There was vague hype and speculation about it months before it was actually released, and in the end we got a model that is objectively impressive when it comes to benchmarks and is objectively better at certain tasks, but otherwise in my mind fell way short of the expectations being set by more "enthusiastic" commentators.

Now, normally if someone is prone to hype and sensationalism on the internet one would typically learn to ignore them and preserve your mental health, problem is that sometimes OpenAI employees (and other employees of frontier labs) can't resist making sensationalist claims themselves, i.e. claiming that we're a couple of years away from AGI or whatever.

On the one hand, it's a smart and reasonable thing to bet on expert advice, on the other hand, the experts are making very bold claims that I just struggle to see coming to fruition. The idea that you could simply scale an LLM until we achieved AGI always felt a little suspect to me.

Curious to see how others have felt about this.

I'm really curious to see how this pricing plays out. I constantly hear on Twitter how certain influencers would be more than willing to pay more than $20/month for unlimited access to the best models from OpenAI / Anthropic. Well, now here's a $200/month unlimited plan. Is it worth that much and to how many people?

As someone who's struggled to really get into podcasts, I'm convinced that most people who enjoy podcasts don't really actively listen to them, they just like that extra bit of noise in the background while they do something else, with the added bonus that they might just listen at just the right time to pick up some interesting factoid.

Don't get me wrong, I like podcasts, Fall of Civilizations being a personal favorite of mine, it's just that my desire to try and actively listen to them requires that I carve out an hour (or much longer in the case of FoC, worth it) of my free-time to eliminate any distractions and focus. Pretty hard to do when there are so many other things I could be doing with my time besides sitting there trying my best to listen.

Anyway, I bring it up because I'm convinced that the people promoting AI-podcasts are mostly made up of the aforementioned people who just listen to them for noise.

Not meeting expectations != not better than the previous models.

The Information reporting was a bit more clear on this. Orion is better than GPT-4, it's just that they were expecting a leap in capabilities comparable to what we saw going from GPT-3 to GPT-4. In other words, they were expecting essentially a GPT-5, and Orion wasn't that good.

It's kind of, I don't know, "weird", observing how there's all these news outlets reporting on how essentially every up-and-coming model has not performed as expected, while all the employees at these labs haven't changed their tune in the slightest.

And there's a number of reasons why, mostly likely being that they've found other ways to get improvements out of AI models, so diminishing returns on training aren't that much of a problem. Or, maybe the leakers are lying, but I highly doubt that considering the past record of news outlets reporting on accurate leaked information.

Still though, it's interesting how basically ever frontier lab created a model that didn't live up to expectations, and every employee at these labs on Twitter has continued to vague-post and hype as if nothing ever happened.

It's honestly hard to tell whether or not they really know something we don't, or if they have an irrational exuberance for AGI bordering on cult-like, and they will never be able to mentally process, let alone admit, that something might be wrong.

I agree that existing benchmarks are no longer useful now that there's basically nothing left in them that seems to stump LLMs.

But when I hear that models are failing to meet expectations, I imagine what they're saying is that the researchers had some sort of eval in mind with room to grow and a target, and that the model in question failed to hit the target they had in mind.

Honestly, problem with sentiments like these is on Twitter is that you can't tell if they're being sincere or just making a snarky, useless remark. Probably a mix of both.

Kind of reminds me of an idle thought I have every now and then. Between the sheer difficulty of establishing any kind of foothold on Mars, and the vast amount of uninhabited land, it’s curious that more thought hasn’t been given into the much easier task of making the empty parts of the planet more bearable.

Alas, the list of reasons to live in the Great Plains is very short, which is also why I’m kind of skeptical of terraforming the American West. You can make existing major cities more livable, sure, but don’t expect a surge of people moving to Montana or Wyoming.

By contrast, Los Angeles and Miami have ocean access. Terraforming coastline is a no-brainer.

Tried my standard go-to for testing, asked it to generate a voronoi diagram using p5js. For the sake of job security I'm relieved to see it still can't do a relatively simple task with ample representation in the Google search results. Granted, p5js is kind of niche, but not terribly so. It's arguably the most popular library for creating coding.

In case you're wondering, I tried o1-preview, and while it did work, I was also initially perplexed why the result looked pixelated. Turns out, that's because many of the p5js examples online use a relatively simple approach where they just see which cell-center each pixel is closest to, more or less. I mean, it works, but it's a pretty crude approach.

Now, granted, you're probably not doing creative coding at your job, so this may not matter that much, but to me it was an example of pretty poor generalization capabilities. Curiously, Claude has no problem whatsoever generating a voronoi diagram as an SVG, but writing a script to generate said diagrams using a particular library eluded it. It knows how to do one thing but generalizes poorly when attempting to do something similar.

Really hard to get a real sense of capabilities when you're faced with experiences like this, all the while somehow it's able to solve 46% of real-world python pull-requests from a certain dataset. In case you're wondering, one paper (https://cs.paperswithcode.com/paper/swe-bench-enhanced-codin...) found that 94% of the pull-requests on SWE-bench were created before the knowledge cutoff dates of the latest LLMs, so there's almost certainly a degree of data-leakage.

As someone who otherwise hates genAI, I must admit, this is actually a very cool demo and a very sensible application of AI.

Kind of thrown off by the Phil Fish comments.

Like, are we talking about the guy who made Fez? That Phil Fish? Is it 2012?

As much of a curmudgeon as I am on AI I do sincerely believe that one area that it's effective at is going from absolute beginner to decent understanding on something completely new and foreign.

Recently I've been somewhat curious about Skia, the graphics library that Google uses. A while back I fielded a few questions about Skia to Claude. Nothing crazy, just questions on how to draw a few primitive shapes, but I felt it would have taken some effort to find the answers on my own.

And I must say I was pretty satisfied with what it gave me back.

A complete tangent, but I think a big reason why I'm kind of dismissive of AI is because people who speculate on what it would enable make it honestly sound kind of unimaginative.

> AI models will soon serve as autonomous personal assistants who carry out specific tasks on our behalf like coordinating medical care on your behalf.

I get it, coordinating medical care is exhausting, but it's kind of amusing that rather than envisioning changing a broken system people instead envision AIs that are so advanced that they can deal with the complexity of our broken systems, and in doing so potentially preserve them.

Related btw to using AI for code.

This is going to be the real test for Waymo. From anecdotal experience, Austin has more inclement weather, and its road infrastructure is lacking in some parts of town.

Yeah, now that you mention it I also see that. It was clearly meant to spawn after 3 seconds. Seems on successive attempts it also doesn't quite wait 3 seconds.

I'm kind of curious if they did a little bit of editing on that one. Almost seems like the time it takes for the squirrel to spawn is random.

The idea of Github having a unique "taste" advantage resonates with me a lot. I don't like the fact that Github is using my code to feed Microsoft's AI ambitions, but I dislike Bitbucket and Gitlab more simply on the grounds that they "don't look fun".

It's tricky, because any serious Github competitor would implicitly have to compete by attracting the deep pockets of enterprise clients, who care little for "fun". Getting revenue from solo devs / small teams is an uphill battle, especially if you feel obliged to make your platform open source.

Still, I wish someone would make a Github competitor that's fun and social.

This is far, far more rigorous than the experiment Microsoft behind Microsoft's claim that Copilot made devs 55% faster.

The experiment in question was to split 95 devs into two groups and see how long it took each group to setup a web server in Javascript. Control took a little under 3 hours on average, the copilot group took 1 hour and 11 minutes on average.

https://github.blog/news-insights/research/research-quantify...

And it is thanks to this weak experiment that Github proudly boasts that Copilot makes devs 55% faster.

By contrast the conclusion that Copilot makes devs ~25% more productive seems reasonable, especially when you read the actual paper and find out that among senior devs the productivity gains are more marginal.