Bjorkbat

Makes me think of the horse browser (https://gethorse.com), namely the fact that unlike pretty much all other browsers it's a paid, subscription product. You actually have to pay $60 a year in order to use it.

Sounds absolutely ridiculous when you consider that you haven't had to pay for a browser since Netscape, and even then I think you only had to pay once for it (it was before my time, might need some help on this statement from people actively using Netscape in its heyday), but this whole Google antitrust thing has made me appreciate just how fragile the current browser status quo happens to be. Safari and Edge are fine, but I don't particularly like using either, and to be frank the reasons for using an open-source browser besides Firefox or Chrome are largely ideological.

It just might be the case where if you want an actually good browser, you'll have to start paying for it.

/vg/ also had a pretty cool amateur game dev general thread (/agdg/). No one was making any hidden gems there, but it wasn't trash either. At any rate, I liked it.

Minor pet-peeve of mine, I really don't like the term "superforecaster". First time I encountered it was in association with some guy who was making predictions a year or two out.

Which to be fair it actually is kind of impressive if someone can make accurate predictions about the future that far head, but only because people are really bad at predicting the future.

Implicitly when I hear "superforecaster" I think they're someone that's really good at predicting the future, but deeper inspection often reveals that "the future" is constrained to the next 2 years. Beyond that they tend to be as bad as any other "futurist".

Can't believe I'm actually rooting for the copyright cartels in this fight.

But that does make me think, that in a sane society with a functional legislature I wouldn't have to pick a dog in this fight. I'd have have enough faith in lawmakers and the political process to pursue a path towards copyright reform that reigns in abuses from both AI companies and megacorp rightsholders

Alas, for now I'm hoping that aforementioned megacorps sue OpenAI into a painful lesson.

I've honestly thought of hacker spirit as embodying a kind of homesteader ethos in a way. There's this homesteading book I bought a long time ago when I was in college, rich with illustrations on how to do everything from raise animals and grow food to building a house, processing lumber, drilling a well, everything. The same fascination I have with homesteading and DIY culture extends into my interest in technology, and I suspect this is the same with a lot of developers as well.

I'm amused by all the flags this article has. It reinforces this belief that "vibe-coding" isn't something that evolved organically, but was forced. I wouldn't go as far as to call it "astroturfed", I believe it was a spontaneous emergence, but otherwise it feels like an effort by a disproportionately small group of people desperately trying to make vibe-coding a thing for whatever reason.

Tangential, but this reminds me of something someone said on Twitter that has resonated with me ever since. Startups targeting developers / building developer tooling are arguably one of the worst startups to build, because no matter how much of a steal the price is relative to the value you get, developers insist they can build their own or get by with an open-source competitor. We're as misguided on value as we are on efficiency and automation (more specifically, the old trope of a dev spending hours to automate something that takes minutes to do).

My first thought seeing this and looking at benchmarks was that if it wasn’t for reasoning, then either pundits would be saying we’ve hit a plateau, or at the very least OpenAI is clearly in 2nd place to Anthropic in model performance.

Of course we don’t live in such a world, but I thought of this nonetheless because for all the connotations that come with a 4.5 moniker this is kind of underwhelming.

I'm going to add a bit of a left-field contribution since his work is less generative coding more mathematics and geometry in general, but it has inspired me when it comes to generative coding. I'm assuming the works he's done which are printed rather than painted were created in such a way not too dissimilar from creating coding.

Anyway, Clark Richert (https://www.clarkrichert.com/, see also MCA's page on him as well https://octopus.mcadenver.org/artists/clark-richert).

Related, if you like startup simulators, then you'll love The Founder (http://thefounder.biz/).

Incidentally Francis Tseng also worked on Half-Earth Socialism, which is also quite fun, and both games happen to be open-source (https://github.com/frnsys/)

Yeah, I have to admit it looks kind of "uncanny", but at the same time it kind of has a sense of personality to it that other humanoid robots lack, in part because it has face with two distinct eyes.

The knit "suit" is also kind of a nice touch. Can't help but think that a robot wearing a sweater is kind of endearing.

Yeah, I saw the pass@7 figure as well, and I'm not sure what to make of it. On the one hand, solving nearly half of all tasks is impressive. On the other hand, a machine that might do something correctly if you give it 7 attempts isn't particularly enjoyable to use.

>But sonnet solved over 25% of them and made 60 grand.

Technically it didn’t since all these tasks were done some time ago. On that note, I feel like putting a dollar amount on the tasks it was able to complete is misleading.

In the real world, if a model masquerading as a human is only right 25% of the time, its reviews on Upwork would reflect that and it would never be able to find work ever again. It might make a couple thousand before it loses trust.

Of course things would be different if they were open and upfront about this being an LLM, in which case it would presumably never run out of trust.

And again, Expensify is an anomaly among companies in that it gives freelancers well articulated tasks to work on. The real world is much more messy.

I think that's a premature conclusion to take from this benchmark.

Something to keep in mind is that Expensify is kind of an anomaly in that it hires freelancers by creating a well-articulated Github issue and telling them to go solve that. This is about as ideal as you can hope to ask for when it comes to articulating requirements, and yet o1 with high reasoning could only solve 16.5% of tasks formatted this way.

Not to mention, these models perform a lot worse than their SWE-bench results would otherwise suggest.

Big picture, there's a funny trend when it comes to generative AI of inflated expectations that rapidly deflate once we use them in the real world. I still remember being a little bit freaked out by o1 when it came out because it scored so well on a number of benchmarks. Turns out, it's worse than Claude Sonnet when it comes to coding. Our expectations are consistently inflated by hype and benchmarks, but then once we use them in the real world we find out that they're not as great as the benchmarks would otherwise suggest.

Kind of feels like this is going to go on forever. A new model is announced, teased with crazy benchmark results, once people get their hands on it they're slightly underwhelmed by how it performs in the real world.

To be fair I'm pretty sure Gary Marcus did in fact bring that up at one point, and if I recall correctly the incident was embarrassing enough for Google to briefly pause the generation of AI images until it could correct the awkward bias in its image generation model.

My belief is that software engineering benchmarks are still a poor proxy for performance on real world software engineers tasks, and that there's a decent chance a new model might saturate a benchmark while being kind of underwhelming.

A simple example, if a human scored 50% on SWE-bench verified, it's fair to say that this person is a very competent software engineer. Popular frontier models like Claude Sonnet and OpenAI's o3 can score 50% on SWE-bench, and can score even higher with special tooling, but compared to an actual human software engineer can't seem to competently perform a lot of programming tasks on their own.

Although, if a model did consistently score more than 99% on various software engineering benchmarks that might be different, as it would imply a very real sense of competence. That's a pretty substantial if though. To my knowledge there isn't a single model out there that can consistently score more than 99% on any given benchmark. The o1 model scored very well on certain MMLU categories, 98.1% on college mathematics, but I'm not sure if this result will continue to hold on a similar benchmark evaluating college-level undergraduate mathematics.

Also, something else to consider, we take for granted how often we're able to perform tasks with more than 99% accuracy and how quickly things would fall apart if this weren't the case. If the average human driver was only able to make an accident-free trip only 99% of the time that would imply that they'd get in a wreck every 100th time they drive their car.

Granted, software engineering might be the exception to this rule, but then again, depends on what you're measuring. When it comes to more-or-less discrete steps, we're arguably pretty good at writing programs that capture our intent, and I could foresee an AI that only gets this right 99% of the time to be a pain in the ass to work with. If a feature ticket requires 10 different sub-tasks to be done correctly, then an AI that can do each sub-task correctly 99% of the time has a roughly 90% chance of doing the whole feature ticket correctly, which is still good but compounded over many feature tickets could be exhausting to deal with. An AI that has only a 90% chance of doing each sub-task correctly would almost certainly fail to implement this hypothetical feature ticket.

Mind you, statistics is not my domain so if there are any errors in my logic please correct me.

This kind of reminds me of when there was a lot of hype around messenger apps and this idea that we'd just do everything through a chat interface / chat bot.

It never panned out, arguably because the technology wasn't quite there yet (this was well before ChatGPT came out), but I thought the bigger problem was that people thought that a chat UI was the ultimate user interface. Just didn't feel right to me. For simple tasks, sure, but otherwise it felt like for "exploratory" tasks it made more sense to have a graphical user interface of some kind.

Same sentiments apply to the hype around agents. Even in a hypothetical world where agents work as well as any human I don't think an agent/chatbot UI is necessarily the ultimate user interface. If I'm asking an agent questions, it makes sense for it to show rather than tell in many contexts. Even in a world where agents capture much of the way we interact with computers, it might make more sense for them to show us using 3rd party SaaS apps.

Before people lose their minds on AlphaGeometry, I thought I'd share this gem the r/math subreddit lending some insight into how the original AlphaGeometry appears to work from the perspective of someone far more literate in math than the rest of us.

https://www.reddit.com/r/math/comments/19fg9rx/some_perspect...

The tldr is that a lot of the heavy lifting was done by an algorithm used called Deductive Database + Algebraic Relations.

I must stress that the results were still impressive, at the time scoring a silver in Olympiad Geometry was seen as something out of reach for AI, and it's impressive that they were able to do this with a mostly deterministic approach. The point is that you really didn't need that much AI to actually score a silver.

Most of my observations have been that people are using it to make personal software. That is to say, software with an intended user base of just yourself and maybe friends and family.

For software meant to be consumed by the masses it's too unreliable for the all the boring details, but otherwise if you want something that serves a specific purpose then sure, it seems to work really well.

Otherwise though I haven't really hard of any non-technical founders leveraging it to finally get their app off the ground.

This kind of reminds me of back when the hype cycle was focused on Messenger apps and the idea of most online behavior being replaced with a chatbot. God I hated the smug certainty of (some, definitely not all!) UX designers at the time proclaiming that chat was the ultimate interface.

Absolutely insane that all the doors unlocked by being able to interact with a computer graphically, and yet these people have visions of the future stuck in the 60s.

So, I still think this is a cool tool for search reasons, but otherwise the tendency to hallucinate makes it questionable as a researcher.

Hypothetically speaking, if the time you saved is now spent verifying the statements of your AI researcher, then did you really save any time at all?

If the answers aren't important enough to verify, then was it ever even important enough to actually research to begin with?

Actually sounds pretty cool, but the graph on expert level tasks is confusing my expectations. Saying it has a pass rate of less than 20% sounds a lot like saying this thing is wrong most of the time.

Granted, these strike me as difficult tasks and I’d likely ask it to do far simpler things, but I’m not really sure what to expect from looking at these graphs.

Ah, but the fact that it bothers to cite its sources is a huge plus. Between that and its search abilities it sounds valuable to me

So am I to understand that they used their internal tooling scaffold on the o3(tools) results only? Because if so, I really don't like that.

While it's nonetheless impressive that they scored 61% on SWE-bench with o3-mini combined with their tool scaffolding, comparing Agentless performance with other models seems less impressive, 40% vs 35% when compared to o1-mini if you look at the graph on page 28 of their system card pdf (https://cdn.openai.com/o3-mini-system-card.pdf).

It just feels like data manipulation to suggest that o3-mini is much more performant than past models. A fairer picture would still paint a performance improvement, but it look less exciting and more incremental.

Of course the real improvement is cost, but still, it kind of rubs me the wrong way.

I have to admit I'm kind of surprised by the SWE-bench results. At the highest level of performance o3-mini's CodeForces score is, well, high. I've honestly never really sat down to understand how elo works, all I know is that it scored better than o1, which allegedly as better than ~90% of all competitors on CodeForces. So, you know, o3-mini is pretty good at CodeForces.

But it's SWE-bench scores aren't meaningfully better than Claude, 49.3 vs Claude's 49.0 on the public leaderboard (might be higher now due to recent updates?)

My immediate thoughts, CodeForces (and competitive programming in general) is a poor proxy for performance on general software engineering tasks. Besides that, for all the work put into OpenAI's most recent model it still has a hard time living up to an LLM initially released by Anthropic some time ago, at least according to this benchmark.

Mind you, the Github issues that the problems in SWE-bench were based-off have been around long enough that it's pretty much a given that they've all found their way into the training data of most modern LLMs, so I'm really surprised that o3 isn't meaningfully better than Sonnet.

I'm not going to say that this isn't creative coding, since that sounds elitist. What I will say is that this is a very shallow interpretation of creative coding.

With very little learning and modest effort you can make very fun 2D art pieces with p5js. If 3D is more your jam, there's three.js. If you hate web dev, there's countless other creative coding libraries and frameworks for whatever language is more your speed, such as Processing for Java, or openFrameworks for C++, etc etc.

But they're not talking about that. Instead they're talking about new browser features like OKLCH and container queries.

Which to be fair they've given some very cool examples of people using browser features creatively, but otherwise these features (or lack thereof) were never the bottleneck for my creativity. The real unlock for me was learning a lot of math, things like simple linear algebra so I can do otherwise complex transformations across a scene, or learning the underlying mathematics of bezier curves.

Like, this is a sanitized, corporate perspective on creative coding, but then again I'm not sure what to expect. This is a Figma blog post about career advice more than anything else.

I'm personally kind of anti-NFT myself, but I have to admit I do like the generative art NFT scene, especially back when Hic Et Nunc was around and people were selling fun little things they made for the equivalent of $5 of TEZ.

To me the thing I hated most about NFTs was the unbridled greed and opportunism, people trying to get rich quick by making bad pixel art or stealing other people's art as their own, so it was really refreshing to see a community of people making things fun things and selling them for dirt cheap, and seeing other people buy said things simply because it was kind of fun.

This is in line with my perspective that writing as a form of self-reflection and thinking out loud is underrated. Likewise, programming as a form of thinking out loud is also underrated.

For complex problems, I find that it doesn't matter how much planning I put into the task, I always find shortcomings in my initial logic and imagination once I get into the details and get involved in countless micro-decisions. Personally, I'd prefer thinking out loud through a programming language vs countless back-and-forth with an AI.

I'm still skeptical on the notion that we can remove the human bottleneck on code because code has verifiable solutions.

It's true only to the extent that there's sufficient test coverage to prevent any unwanted side effects. Easy to do with straight forward problems, far more difficult with more complex as well as open-ended problems.

Glad someone brought this up.

I'm personally fine with o3 being tuned on the train set as a way to teach models "the rules of the game", what annoys me is that this wasn't also done with the o1 models or r1. It's a misleading comparison that suggests that o3 is a huge improvement over o1 when in reality much of that improvement may have simply been that one model knew which game it was playing and the others didn't.

I have to admit it's really hard for me to take the noise about agents seriously because they keep bringing up using it for things that don't feel like chores to me, but are actually kind of delightful. I like the anticipation and excitement of planning for the big holiday. I like shopping online for new books or clothes or what have you.

Suppose for a moment that agents worked 100% flawlessly (suspend whatever disbelief you have for a moment and roll with it) a large chunk of the web is still, actually, quite fun for quite a lot of people. Even if the information on the website is just as accessible through a chat interface as it is through a web browser, the website is still probably cooler and has a more fun experience vs using a chat interface.

Obviously not all websites are like this, but the ones I care about are.