deepdarkforest

> 1) Raw inference speed matters more than incremental accuracy gains for dev UX—agree or disagree?

I know you are trying to generate some controversy/visibility, but i think if we are being transparent here, you know this is wrong. People prefer using larger (or reasoning) models, with much bigger diff in tok/sec just for quality in coding, it comes first. Even if i have a big edit to apply, like 5k tokens, 200-300ms of difference in edit time are nothing. Edit speed is definitely not a bottleneck for dev UX, quality is. A dev who wants to save 200ms every code change over quality is someone who well, i cannot relate. If im using 1-2 agents in parallel, most of the time the edits are already applied while im reviewing code from the other agents. But again maybe that's just me.

Speaking of quality, how do you measure it? Do you have any benchmarks? How big is the difference in error rate between the fast and large model?

It's not silly. I would bet 99% of the users don't care that much to do that. A hardcoded regex like this is a good first layer/filter, and very efficient

Cursor raised 900M, are losing market share to claude code(resorting to poaching 2 leads from there [1]), AND they're decreasing the value of their product? Huge red flag. They should be able to burn cash like no tomorrow. Also, the PR language on this post, and the timing(midnight on a US holiday) is not ideal.

This news coupled with google raising the new gemini flash cost by 5x, azure dropping their startup credits, and 2-3 others(papers showing RL has also hit a wall for distilling or improving models), are now solid signals that despite what Sam altman says, intelligence will NOT be soon too cheap to meter. I think we are starting to see the squeeze from the big players. Interesting. I wonder how many startups are betting on models becoming 5-10x cheaper for their business models. If on device models don't get good, I bet a lot of them are in big trouble

[1] https://www.investing.com/news/economy-news/anysphere-hires-...

From a cash flow perspective of course it makes sense to sell the future before you have it as working product. It just needs a great salesman or narrative to keep it going, im not arguing that.

> that feedback loop is very important for improving machine learning models at scale

Oh will you have your own feedback loop with let's say user's data? Or you meant as an example?

> * That's why we're open sourcing everything, to avoid the feeling of overpromising on what is ready today*

I agree here, it helps the today, but I dont think it helps the feeling of overpromising on what is ready today, its more like, even if it's open source , it does not increase the chances of it being ready/autonomous in the future. (im just playing devils advocate here)

I also agree with the intermediate benchmarks for sure, this is more to what i was referring to, it would be nice to see some more short term usecases/fun applications that are realistic to hit today or in the nearer future, that would drive a lot of sales value, at least for me, rather than go from now to full autonomy. Good luck!

> the demand side, the basic problem with humanoid robots is that they're mostly useless right now ... ... to square this circle, ... we will provide free hardware and software upgrades until we are able to make the robot fully autonomous...This way, we can have some extra cash upfront to kickstart development

Congratulations guys! The technical stuff is above my paygrade, but you have a cracked team and with open source you will have a great chance to be close or at SOTA level at your price point.

However, it looks to me that your core thesis is yes, when the autonomous robots get good enough, even at a medium family car price range they will sell like candies. Sure. But since you also want to have the cash now, to who exactly are you selling? Yes you promise that you will support the full autonomy option, but this sounds weirdly similar to Tesla selling cars promising the FSD, which we all know how that story went.

I'm not saying you won't deliver, I'm just saying you might need to a bit more careful in your story selling/narrative for this. For example, i would be super interested to get one for like 2k if it's not useful now, but paying 10k for essentially promises and possible upgrades is a bit iffy. Hence i would like to at least see some plug in and play current usecases? Even if they are just for fun.

If you actually click on the link, it mentions this both in the abstract, and a detailed comparison of evidence in a whole table.

The main problem with these approaches is that most sites now are useless without JS or having access to the accessibility tree. Projects like browser-use or other DOM based approaches at least see the DOM(and screenshots).

I wonder if you could turn this into a chrome extension that at least filters and parses the DOM

the difference is simple, addiction is the compulsive/habitual action of something that does not have positive effects, more likely negative than neutral. For example, being "addicted" to gym is fine, because even if you enjoy the dopamine after, not the actual exercise, it's still good for you overall, and you cannot really overdo it. Caffeine makes you more productive, and you cannot really spiral, hence why caffeine is legal, but opioid drugs are not.

But for tech, scrolling tiktok for 5 hours a day, definitely does not have positive outcomes. It's like gambling. It's dopamine hit for dopamine hit's sake, and the delivery system is actively harming you.

This might not get a lot of traction because it's very technical, but i wanted to say a massive well done for the effort. 20k words on anything this specific is not a joke. I wish i would put this level of commitment to anything in life, this was inspiring if nothing else.

What irks me about anthropic blog posts, is that they are vague about details that are important to be able to (publicly) draw any conclusions they want to fit their narrative.

For example, I do not see the full system prompt anywhere, only an excerpt. But most importantly, they try to draw conclusions about the hallucinations in a weird vague way, but not once do they post an example of the notetaking/memory tool state, which obviously would be the only source of the spiralling other than the SP. And then they talk about the need of better tools etc. No, it's all about context. The whole experiment is fun, but terribly ran and analyzed. Of course they know this, but it's cooler to treat claudius or whatever as a cute human, to push the narrative of getting closer to AGI etc. Saying additional scaffolding is needed a bit is a massive understatement. Context is the whole game. That's like if a robotics company says "well, our experiment with a robot picking a tennis ball of the ground went very wrong and the ball is now radioactive, but with a bit of additional training and scaffolding, we expect it to compete in Wimbledon by mid 2026"

Similar to their "claude 4 opus blackmailing" post, they intentionally hid a bit the full system prompt, which had clear instructions to bypass any ethical guidelines etc and do whatever it can to win. Of course then the model, given the information immediately afterwards would try to blackmail. You literally told it so. The goal of this would to go to congress [1] and demand more regulations, specifically mentioning this blackmail "result". Same stuff that Sam is trying to pull, which would benefit the closed sourced leaders ofc and so on.

[1]https://old.reddit.com/r/singularity/comments/1ll3m7j/anthro...

By competitive, i mean no.1 in LM arena overall, in webdev, in image gen, in grounding etc. Plus, leading the chatbot arena ELO. Flash is the most used model in openrouter this month as well. Gemma models are leading on device stats as well. So yes, competitive

> Sundar is a really uninspiring leader

I understand, but he made google a cash machine. Last quarter BEFORE he was CEO in 2015, google made a quarterly profit of around 3B. Q1 2025 was 35B. a 10x profit growth at this scale well, its unprecedented, the numbers are inspiring themselves, that's his job. He made mistakes sure, but he stuck to google's big gun, ads, and it paid off. The transition to AI started late but gemini is super competitive overall. Deepmind has been doing great as well.

Sundar is not a hypeman like Sam or Cook, but he delivers. He is very underrated imo.

remember, gaps in the market sometimes exist for a reason. Forget AI. How many open source, community driven and privacy first browsers have made serious money?

Brave is a decent example but their business model is actually complicated, it includes a lot of little stuff. And they dont have the unit cost of LLMs (im assuming at some point you will take the burden of the llms, if not local)

This is definitely a winners take all market. Kudos for giving it a shot, but imo browser projects are just too big for a team of 2/3. Plus, google has already demoed at IO the first hint at this. IMO you just cannot move fast enough to grab enough market share as a first/second mover before google just does it on chrome, and that's assuming you can outcompete with Dia in the first place. Even browser-use can do this also, and they have good distribution already.

Good luck, but in your place i would at least start with something that a certain ICP needs more. Many, many manhours have been wasted by ambitious technical founders on taking down Chrome. (many also starting from a chrome fork itself). But none of them succeeded. We only have limited energy

you will run into the same problems the extension approaches have (Like nanobrowser etc). Which is if i have to supervise constantly for non reversible actions, then im no more efficient(actually less i would argue) than just doing the task myself. A human in the loop- pause just before a non reversible action asking for approval maybe? And the user can see all the current things that need approval. But a stop button works just too late imo

For easy/format stuff for specific journals it will be useful. But please, please for the love of god don't try to give actual feedback. We have enough GPT generated reviews on openreview as it is. The point of reviews is to get deep insights/comments from industry experts who have knowledge ABOVE the LLMs. The bar is very low, i know, but we should do better as the research community.

Terrible stuff and a reddish flag. First of all, gpt signs all over the blog post, reads like a bottom of the barrel linkedin post.

But more importantly, why double and triple down on no RAG? As with most techniques, it has its merits in certain scenarios. I understand getting VC money so you have to prove differentiation and conviction in your approach, but why do it like this? What if RAG does end up being useful? You'll just have to admit you were wrong and cursor and others were right? I don't get it.

Just say we don't believe RAG is as useful for now and we take a different approach. But tripling down on a technique so early into such a new field seems immature to me. It screams of wanting to look different for the sake of it.

wouldn't that be just 3rd party llms.txt?

just wanted to say this was the most relatable take i have read so far, and i've read a lot. Exact same experiences. And you didnt even touch on the MCP's that enable these things to go wild as well. I think our takes are not being taken seriously for 2 reasons.

First marketing gaslighting from the faangs and hot startups with grifters that managed to raise and need to keep the bullshit windmill going.

Second is that these tools are relatively the best in boilerplate nextjs code that the vibecoders use to make a very simple dashboard and stuff, and they're the noisy minority on twitter.

There is basically zero financial incentive to admit LLM's are pushed dangerously beyond their current limits. I'm still figuring a way to go short this, apart from literally shorting the market.

1. Oh yes right. I remember trying it out thinking it was going to be brittle because of analytics etc but it filters for those surprisingly well.

2. We are working on https://www.launchskylight.com/ , agentic QA. For the self onboarding version we are using pure CUA without caching. (We wanted to avoid playwright to make it more flexible for canvas+iframe based apps,where we found HTML based approaches like browser-use limited, and to support desktop apps in the future).

We are betaing caching internally for customers, and releasing it for the self-onboarding soon. We use CUA actions for caching instead of playwright. Caching with pixel native models is def a bit more brittle for clicks and we focus on purely vision based analysis to decide to proceed or not. I think for scaling though you are 100% right, screenshots every step for validating are okay/worth it, but running an agent non-deterministically for actions is def an overkill for enterprise, that was what we found as well.

Geminis video understanding is also an interesting way to analyze what went wrong in more interactive apps. Apart from that i think we share quite a bit of the core thinking, would be interested to chat, will DM!

Very cool. 1) How do you deal with timings? If a step includes clicking on a link or something that needs loading, then if you just fire off the generated playwright code at at once, some steps might fail because the xpath is not there yet. So to be safe, i'm guessing you would have to wait using the difference in the timestamps in the json. 2. For self healing, we worked on something similar, and we found it's very easy to get off the rails if one step fails because if your assertions of if the fix was correct are off, then the next and next steps will also fail etc. The most stable way was to just regenerate all steps if a step fails in 2 attempts (2 runs of the flow) consecutively. If the xpath is broken for a step, very likely the subsequent ones won't be worth healing individually.

Not sure if this can work. We played around with something similar too for computer use, but comparing embeddings to cache validate the starting position is super gray, no clear threshold. For example, the datetime on the bottom right changes. Or if it's an app with a database etc, it can change the embeddings arbitrarily. Also, you must do this in every step, because as you said, things might break at any point. I just don't see how you can reliably validate. If anything, if models are cheap, you could use another cheaper llm call to compare screenshots, or adjust the playwright/api script on the fly. We ended up writing up a quite different approach that worked surprisingly well.

There are definitely a lot of potential solutions, I'm curious where this goes. IMO an embeddings approach won't be enough. I'm more than happy to discuss what we did internally to achieve a decent rate though, the space is super promising for sure.

Looks super interesting. I have couple of very annoying workflows that i'm looking to automate but no API's. Can i install my own apps?

Love when someone that actually knows their stuff does a deep breakdown like this. Super useful

I wonder though if all it matters is the last punchline. The profit margin vs competitors. If llms truly get commoditized and do not benefit from economy of scale(like infra/aws) because they would be run in house, then would all companies that use them be in a nash equilibrium? Like everyone has access to the same stuff, same reasoning, same benefits. The only people who will really win in enterprise are consultants that find the best way to leverage and implement agents, not the model makers themselves. And of course data will still be king

Very clean writeup. On the attention sinks, you mention they enable "infinite-length sequence processing". What does that mean exactly in practice? Isn't deepseek still capped at 128k?