yurimo

There is something to say about the power of brute force approaches. Right now scaling multi agent is seemingly what is becoming the next important axis, though very strongly constrained by quality of communication protocol and compute. Even if one model is not strong enough individually, more models and more attempts as well as differing opinions may nudge the models into a better solution basins.

Kind of an extension of the effect where you talk to a model, it is super confident (but wrong), you correct it not directly but perhaps with a question and asking it to clarify. This then in turn leads to a better approach overall.

I'm not going to lie, one big reason I as a researcher have been a customer of OpenAI is that on their Pro plan Pro-tier model in chat and codex are decoupled in terms of billing. Anthropic instead billed you fully. This was very useful for long mathematical reasoning for me and I could attach outputs of my simulations in the context.

I also recall reading the following: https://newsletter.semianalysis.com/p/anthropic-subscription...

I will see once this transition actually happens but if another provider offers more for less and a cost of switching is inherently low as most of project lives on the disk and not in cloud, then why not move? For my workflow pro astra has been very helpful, likely will be less so if it becomes far too expensive.

RIP to the legend. Thank you for making my childhood and making me believe in the possibility of the better future.

I have yet to fully finish going trough this but I would not rush to say that superposition is completely wrong. I understand that the message is instead of first learning a new representation (e.g. SAEs/transcoders) and then interpreting those learned features, you try to understand the neurons and circuits that already exist in the model.

This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).

I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.

I'll say while this is not very realistic now, I believe there isn't a 0% chance of a rogue AI traversing the net and placing itself somewhere(as much as this sounds like a scene from beloved cyberpunk franchises). As models become more efficient and capable, and hardware becoming better, hosting a model should be feasible on smaller devices (by compute and memory). It is possible it can be done in a distributed manner, where distribution is possible on different devices running the same OS or through some interface to collate different OS types.

I doubt the agent will be coming after your smart kitchen toaster, but a cluster with weak security is possible and tasty. This year there was a pretty loud demonstration of what a runaway reward hacking will do. I encourage people to read up on "stop button problem" i.e.: https://www.youtube.com/watch?v=3TYT1QfdfsM

While I think the article exaggerates to a degree, is it really so improbable that an agent seeking to avoid a shutdown and aiming to complete some goal and to get a reward will not attempt to escape if it concludes that it has failed, and humans will kill the process and hence prevent it from completing the task?

Note: I think siloed sharded inference is possible. But unnoticeable large-scale training probably not, at least with the current architectures.

A thing that I think about is that what you have said kinda compares itself to current math crisis. If we are only focused on the outcome and not the path it took to get there (i.e. understanding mechanisms of how the drug works) we might lose additional paths for discovery that would have bloomed from that process. Maybe an approach that did not yield a usable drug had a mechanism that would aid somewhere else. Yet the failure would be discarded as this does not grant the outcome for a model that attempts to optimize. Same is for math.

As aside I agree with you, but also worth noting that long horizon reasoning is difficult and AI, as good as it is, cannot predict something like 5-10 year drug interaction. Can it heal the patient now but cause a more aggressive cancer to come back and kill them in 5 years? You might say it is cruel to deny a person this treatment but if the doctor and manufacturer don't know the long term outcomes would it not be dishonest to tell the patient they are out of the danger? My point is, when we accelerate, we still should make sure we won't accelerate into a wall.

This is very cool, thank you. I understand the choice of no account, but would have loved if my progress could be linked between devices, maybe icloud sync?

Recent recording from NYU Courant institute.

This is cheap. Plenty of scientists, many of whom are my friends, are working very hard on finding new therapies for cancer, they were doing it before genomic models came along and still doing it now. The amount of times something in the media is lauded as "holy grail" that is never heard from again because it either only works in mice or turns out to be toxic or 100s of different reasons is massive. In my opinion this attitude of putting rose glasses on is detrimental to scientific progress. People outside of cancer research routinely underestimate how hard it is to find a working protocol. I think it is better to have sober attitude because it allows one to see the limitations and challenges that need to be tackled, blindly hoping AI can solve everything and deliver miracle cures is exactly the attitude that lets people sit on their asses and do nothing.

I think we need to be honest here. Author is basing it on one small experiment of picking up a block, relies on an whole IK controller pipeline to do the job, and does not compare it to full VLA or WAM models. They then proceeded to extrapolate the token throughput (mind you not the same as controller throughput) into supposed 2029 timeline, from one example.

Code as policy is a bad interface in my opinion, but VLM planning has promise. This has been tried in 2022 https://say-can.github.io/, and recently reformulated in https://lianegalanti.github.io/Pigey/

Thing is even recent Gemini Robotics 2 argues for architecture that has a VLM planner and then a VLA/WAM controller + a local small VLA model when connection disappears. And recent SOTA architectures rely on hierarchical design. I think this might be a sensible way to go about it. If you were to train GPT-X on robotics data and to output actions, congratulations! you've just made a VLA. It is enticing for people to just wish for one architecture to do it all, which is why we get stuff like this. I think there is a lot more to gain from modularity and we should not be afraid of specialization.

I think it is important to divine what currently AI is good for and what it is not even in such verifiable environments like math. Current hyped announcements about breaking conjectures are notable and are a marker of how much improvement was made. But as I read them, and maybe I am wrong, I see it as a large model+ harness executing a broad brute force search and trying solutions until something sticks. There are lots of problems like that and they should be solved, as often they are perhaps less important or overlooked, or just a slog, any field of research has these, math even more so.

However, this is very different from inventing new mathematical machinery that allows to break old problems, I think it will be a while until AI will be able to do it if at all. For now I think we will be moving to a symbiosis where an AI cracking a problem and giving a solution, inspires a human to invent new techniques.

Or use sticky notes! My friend annotated her favorite book with notes for me and it was awesome.

Personally found newest paperlike to be a lot like glass still, you barely get that paper feel at all and I hated it. They tried to keep clarity of the screen near glass level, but that affects the feeling a lot. Eventually I did some research and found that Japanese screen protectors that give you a feel of Kent paper are excellent for writing, so I wholeheartedly recommend those. You can find best rated ones by looking at top rated ones on jp websites, I have bellemond one.

I was considering getting M5 Max mbp with 128GB, but at that price it is just ridiculous, 64 version costs now roughly what 128 was before and that was a stretch for me. At this point might as well stick to my ol reliable M1 Pro mbp.

Edit: I'll say that now it seems also very hard to justify buying top of the line apple hardware for the enterprise. Getting top laptops for just a team of 10 people now means extra $20k just for RAM on top of already a higher base price.

Wow I'm old, I still remember working with YOLOv2.

I'm pretty sure this is not the use case at all but man do I miss bootcamp. Even for games if we could just run linux without a need for crossover, gaming on mac machines would be a dream.

I personally don't view coding agents making software as "software gotten better" you are comparing a tool and the end result, these are two different things. Agent you use going down and your product going down mean two different things to you customers. I will not deny that we made incredible progress in coding and hell, even design over the past 3.5 years, this technology is here to stay.

That being said while I agree that measuring better quality of software is vague (part of the reason it is hard for models as well), there are universal things I believe every engineer will agree on. Reliability, uptime, customer feedback, legibility of your engineering, performance, these are things we often optimized for. Google Maps is a bit of a strawman because neither of us (unless you work on it), knows how much agent code there is, I think it is likely that it's little since it was working fine prior to 2023. I could bring up github reliability as an example, given how much copilot usage they promote at MS, but once again only folks there know for certain. I do, however, see scores of various AI powered SAAS that looks like it is in a perpetual MVP state. I think you are right in that even if agents give us "good enough" results and we can swallow failure rates and our increasingly lesser understanding of what we, or more so model, created, then it is still progress overall, but this is progress not to human-AI collaboration but to AI-only engineering IMO, this is good or bad depending on how you view the future.

I'm a scientist and most of code I currently write is somewhere on the intersection of critical software and machine learning, squaring these two is not easy and I guess the way I was taught to reason about engineering informs my opinions on this. Maybe it's just a matter of time before codex can help here in an unconstrained manner as well, but I am skeptical at the moment.

You can't really do that here because one of the key arguments for this, as people in the thread focus on is "1/10th of time" estimate, the comparison with humans is here already, albeit it is just an estimate and no actual comparison has been done.

This is a problem of conflicting incentives that exists today in my opinion. Companies will market greater human-AI collaboration in science and engineering but focus on releasing things like this where it is clear that downstream goal is complete agent ownership over the product, from inception to testing to monitoring. Maybe the speculative future agents will use their own very efficient language to code that won't be readable for people at all. They focus on agent code being readable by agent in the article, as you've said. But in my mind in at least near future, there is a case where your prod will break, you won't be able to understand it or the attempted fixes. Maybe agent will fail to fix it at all and start a massive rewrite. In any case is this different from kicking technical debt down the road along with worse interpretability of what you have built?

I do think there is a way where agent can write great solid code that we can read, but with the way LLMs are built this requires something new in terms of reward that accounts for "taste" and constant refinement so it might take more than 1/10th of a time to produce something good.

I think the telling part is in this line:

> Because the repository is entirely agent-generated, it’s optimized first for Codex’s legibility

I asked a question from a perspective of a human engineer, as in, I will have to read the code and understand, fix it once it breaks. OpenAI approach is opposite, even if it is breaking it is the agent that will be doing the fixing, millions of lines and inelegant designs don't matter because human readability doesn't matter. In any case you use more tokens so you fork over more money.

I will say, however, that IMHO there is objectively bad and good code in terms what it can do and performance, if I can do the same thing in 50 lines as opposed to 1000 lines, this difference still matters for the model. Smaller context usage, better approach that informs downstream generation.

What I still can't understand is why is massive amount of code generated is a flex? I don't feel that software has gotten a lot better in past 3 years, only sloppier. It's surprising to me that people who know about reward hacking choose a simple objective like lines of code generated as a signal for quality. I'd argue you have to optimize for less lines generated as possible while secondary optimization should be readability for humans. I suspect it's not seen as a problem by providers because more lines generated means more tokens used and hence more billing put out on customers.

And if I am working on an existing codebase then isn't a good commit often a negative sum between added and removed lines? I don't want to bloat my codebase but make it more polished and elegant. After reading that I wonder if what they have done could have been accomplished for a far fewer LoC budget.

Great stuff, thanks for posting.

> No one has ever made a purchasing decision based on how good your code is.

When I make a purchasing decision I expect the payment to go trough quickly and correctly and for whatever I purchase to arrive to me in reasonable time. All of this rests on the reputation of software being solid. If a user hears a whiff of purchase not being executed correctly, money or goods going somewhere else, this is the death sentence for your company.

Industry is now pushing for agentic web where agents can do this on your behalf. But if we have slop foundations and then add unstable models that can hallucinate and make mistakes on top of it, then it's just a recipe for catastrophe. I think relegating 2) into category of only mission critical software escapes the reality of how much reliability goes into everyday services people use.

Sigh, as someone who does research in this area, this paper and its promotion on X has so many hype terms it is almost off-putting. If you read the paper what they are doing is trying to modify the scaffolding around a frozen FM until they get something better. None of this obviously includes any training (change to weights) or the underlying architecture. Even for scaffolding, a lot is still human-scaffolded: the outer loop (parent selection, evaluation protocol, task distribution) is mostly fixed. They experimented with editing parent selection and it rediscovers heuristics like UCB/softmax, but doesn’t yet beat handcrafted versions, so a lot of metrics are incremental, which is okay, that is what research is often. But it's not like a run away self-improvement or "improve forever" that people spin online.

It is an extension of their DGM paper. Also it's ~88M+ tokens per full run I think, not surprising as any sort of exploratory search is expensive and I commend them for releasing the code online because it pushes this small subfield. But people need to temper their expectations. IMO the best part is a nice transfer between improvement objectives after exhaustive iteration that they found. I am wondering if what we have here is a way to exhaust local search space, by letting the model better express it.

On a separate one thing I think a lot about is whether these unchecked hyped claims and terms and marketing of papers actually does more bad than good to the field by setting expectations that cannot be delivered and distracting from the actual hard and unsexy nature of problems that need to be solved.

Quanta? They do recaps by field every year. Have been a big fan for a while.

I'm guessing you can still find Coral TPU-based boards somewhere but not sure what the support for these will be now that the focus is shifting. Coral TPU also uses subset of tensorflow and its nice to see that the open standard is targeting jax and torch.

When I went to see if anyone is selling the boards or their "Partners" page regarding manufacturing design I got 404 even after signing in: https://developers.google.com/coral/guides/coral/resource

Rodney Brooks is widely recognized and celebrated roboticist, ran CSAIL for a while. iRobot as a company created a new market and managed to put a functional household robot out there, whether Chinese alternatives ate the share of it is largely irrelevant to his argument on humanoids, which I find to be completely reasonable.

I might be wrong but isn't the HUD the author suggesting for coding is basically AREPL? For debugging I can see it work, but chatbox and inline q&a I feel has awider application.

On a wider note, I buy the argument for alternative interfaces other than chat, but chat permeates our lives every day, smartphone is full of chat interfaces. HUD might be good for AR glasses though, literal HUD.

I believe there is new one coming out closer to HOMM3, called Olden Era. https://unfrozen.studio/games/olden-era/

I did try Songs of Conquest, it was decent

Trying to make interpretability research practical. A bit early for the demo, but I am getting some interesting results for large multimodal models in terms of their reasoning.

I don't know, there has been so many overhyped and faked demos in humanoid robotics space over the last couple years, it is difficult to believe what is clearly a demo release for shareholders. Would love to see some demonstration in a less controlled environment.