deepdarkforest

The Chinese are doing what they have been doing to the manufacturing industry as well. Take the core technology and just optimize, optimize, optimize for 10x the cost/efficiency. As simple as that. Super impressive. These models might be bechmaxxed but as another comment said, i see so many that it might as well be the most impressive benchmaxxing today, if not just a genuinely SOTA open source model. They even released a closed source 1 trillion parameter model today as well that is sitting on no3(!) on lm arena. EVen their 80gb model is 17th, gpt-oss 120b is 52nd https://qwen.ai/blog?id=241398b9cd6353de490b0f82806c7848c5d2...

1. I was not talking about official MCP servers, those are often even free. Im talking about pricing of other devtools for aggregating tools/mcp's. I think this is an obvious space to build i agree, i just worry about differentiation. Its a search space not as big as web search, or complex (order doesnt matter).

2. Yes, i see, this is what i meant by agentic search. Essentially is a tiny subagent, taking list of tools in and out the relevant ones. Still implementable in 5 mins. But i guess if the experience is very smooth enterprise might pay?

1. Oh okay great, maybe clarify it in the pricing page? That mcp server call means just execute. But its' still 10x more expensive right?

2. From what i understand it's just nested search right? It is not anything different, if you do flat or embedding search or fuzzy/agentic nested is a choice for sure, but Im just saying not sure how defensible this is, if all other mcp competitors or even users themselves put in a nested search tool

1. Interesting approach, but the pricing seems 1-2 orders of magnitude too expensive. For your example for slack, It contains 4 calls for an action. Pricing shows 100 dollars per 10k cals, so 1 cent per call. This means, for an agent that lets say does 4 actions, so and your examples show at least 3-4 api calls per action , it's already 12 cents? Similar tools like composio.dev have 200k calls for 29 dollars, so around 70x cheaper (both for the cheapest tier). Even with needing only 1 call for subsequent calls, 1 cent per single api call sounds wrong, at least for our use case it does not economic sense to pay 5-10 cents on top of llm costs on every user query. Apologies if I'm missing something!

2. Could this not be replicated by others by just handmaking a fuzzy search tool on the tools? I think this is the approach that will win, even with rag for lets say 10k plus tools maybe in the future, but not sure how much differentiation this is in the long term, i've made this search tool myself a couple of times already

Of course not! But usually, you can quantify metrics for quality, like uptime, lost transactions, response time, throughput etc. Then you can have accountability, and remediate. Even for other bugs, you can often reproduce and show clearly the impact. But in this case, other than internal benchmarks, you cannot really prove it. There is no accountability yet

Wow. Sneaky. They do not even state the rate of impact for the XLA bug afaik, which affected everyone, not just claude code users, very vague. Interesting.

Claude code made almost half a billion so far[1] (>500m in ARR and its like 9 months old) , and 30% of all users have been impacted at least once, just from the first routing bug. Scary stuff.

Their post mortem is basically "evaluations are hard, we relied on vibe checking, now we are going to have even more frequent vibe checking". I believe it was indeed unintentional, but in the future where investor's money wont come down from the skies, serving distilled models will be very tempting. And you can not be liable to any SLA currently, it's just vibes. I wonder how enterprise vendors are going to deal with this going forward, you cannot just degrade quality without client or vendor even being able to really prove it.

[1][https://www.anthropic.com/news/anthropic-raises-series-f-at-...]

Looks like it’s time to go outside and touch some grass again

They will probably also release sonnet 4.2 or something soon to make people jump back again to try it and hopefully restick

Congrats! Doesn't replit have an integrated database as well? Lovable has supabase, and I'm pretty sure Base44 as well, plus other agent integrations.

That and consumer robotics. The latter will explode if (big if) RL and llm reasoning get combined into something solid. Lots and lots of smart people are working on it already of course, we are seeing great improvements but nothing really usable. i think we will finally get to a real hype stage in maybe 3-4 years

Here is the secret:

Step 1: Vibecode 5 trash generic apps (eg AI interior designers, gpt wrappers)

Step 2: Launch with paying 15k in google/meta ads

Step 3: Receive back ~5k in revenue

Step 4: Spam on twitter+linkedin clickbaity "heres how i reached 60K ARR in 3 milliseconds" posts to attract ex-crypto bros and hustlers

Step 5: Use your newfound following for sponsored content, courses and hopefully actual organic growth for your aforementioned trash apps

Congrats! I think the space is very interesting, I was a founder of a similar windows CUA infra/ RPA agents but pivoted. My thoughts:

1) The funny thing about determinism is how deterministic you should be when to break, its kind of a recursive problem. agents are inherently very tough to guardrail on an action space so big like in CUA. The guys from browser use realized it as well and built workflow-use. Or you could try RL or finetuning per task but is not viable(economically or tech wise) currently.

2) As you know, It's a very client facing/customized solution space You might find this interesting, it reflects my thoughts in the space as well. Tough to scale as a fresh startup unless you really niche down on some specific workflows. https://x.com/erikdunteman/status/1923140514549043413 (he is also building in the deterministic agent space now funnily enough) 3) It actually is annoyingly expensive with Claude if you break caching, which you have to at some point if you feed in every screenshot etc. You mentioned you use multiple models (i guess uitars/omniparser?), but in the comments you said claude?

4) Ultimately the big bet in the RPA space, as again you know, is that the TAM wont shrink a lot due to more and more SAP's, ERP's etc implementing API's. Of course the big money will always be in ancient apps that wont, but then again in that space, uipath and the others have a chokehold. (and their agentic tech is actually surprisingly good when i had a look 3 months ago)

Good luck in any case! I feel like its one of those spaces that we are definitely still a touch too early, but its such a big market that there is plenty of space for a lot of people.

> It handles complex coding tasks with minimal prompting...

I find it interesting how marketers are trying to make minimal prompting a good thing, a direction to optimize. Even if i talk to a senior engineer, i'm trying to be specific as possible to avoid ambiguities etc. Pushing the models to just do what they think its best is a weird direction. There are so many subtle things/understandings of the architecture that are just in my head or a colleagues head. Meanwhile, i found that a very good workflow is asking claude code to come back with clarifying questions and then a plan, before just starting to execute.

Using LLMs as a critic/red teamer is great in theory, but economically is not that more useful, doesnt save that much time, if anything, it increases the time because you might uncover more errors or think about your work more. Which is amazing if you value quality work and you have learnt to think. Unfortunately, all the VC money is pushing the opposite, using LLMs to just do mediocre work. No point of critiquing anything if your job is to output some slop from bullet points, pass it along to the reader/recipient who also uses LLms to boil your slop down back to bullet points and pass it again etc. Even mentally, it's much more enticing or addicting to use LLMs for everything if you don't' care about the output of your work, and let your brain atrophy.

I also see this in a lot of undergrads i work with. The top 10% is even better with LLMs, they know much more and they are more productive. But the rest have just resulted to turning in clear slop with no care. I still have not read a good solution on how to incentivize/restrict the use of LLms in both academia or at work correctly. Which i suspect is just the old reality of quality work is not desirable by the vast majority, and LLMs are just magnifying this

I mean isn't that amazing for an 1 year old product? If it's already better than a terrible dev with an LLM, or better than a decent dev without an LLM, it's not hard to imagine in 2-3-5 years Devin is better and cheaper than most hires you could do. Without having to do HR, equity etc.

This is a shockingly fresh idea. I get that this generates every pixel from scratch, unlike Gemini approaches. But, i wonder how do you think this type of neural OS would be able to communicate with the internet or other similar neural os. It would have to at least send and get http responses?

> The problem is in strcpy in the src files of curl.. have you seen the exploit code ??????

The worst part is that once they are asked for clarifications by the poor maintainers, they go on offense and become aggressive. Like imagine the nerve of some people, to use LLMs to try to gaslight an actual expert that they made a mistake, and then act annoyed/angry when the expert asks normal questions

I don't disagree, but the current sentiment i was referring to seems to be "maximize AI code generation with tools helping you to do that" rather than "prioritize code quality over AI leverage, even if it means limiting AI use somewhat."

This is a launch by a YC company that converts enterprise cobol code into java. Maybe it's my fault, but i tried every single coding agent with a variety of similar tools and whenever i try to parallelize, they clash while editing files simultaneously, i lose mental context of what's going on, they rewrite tests etc.

It's chaos. Thats fine if you are vibe coding an unimportant nextjs/vercel demo, but i'm really sceptical of all this stance that you should be proud of how abstracted you are from code. A kanban board to just shoot off as many tasks as possible and just quickly read over the PR's is crazy to me. If you want to appear a serious company that should be allowed to write enterprise code, imo this path is so risky. I see this in quite a few podcasts, tweets etc. People bragging how abstracted they are from their own product anymore. Again, maybe i am missing something, but all of this github copilot/just reviewing like 10 coding agents PR's just introduces so much noise and slop. Is it really what you want your image to be as a code company?

It's very funny how many layers of abstraction we are going through. We have limited understanding of how LLM's work exactly and why. We now do post training with RL, which again, we don't have perfect understanding of it either. Then you stack LLMs calls and random tools, and you have agents, and you are attempting to benchmark those. (and this exclude voice, computer use agents etc).

It's all just vibes,there is no good general benchmark for agents and i think it's just impossible, there are just way too many degrees of freedom to achieve anything useful. They're just a complicated tool to achieve things. It's like trying to make a general use benchmark of a stack of 10 microservices together. It does not make sense, it just depends on your usecase and your own metrics

No worries.

GGUF is more suitable for the latest open-source models, i agree there. Quant2/Q4 will probably be critical as well, if we don't see a jump in ram. But then again I wonder when/If mediapipe will support GGUF as well.

PS, I see you are in the latest YC batch? (below you mentioned BF). Good luck and have fun!

[flagged]

I don't know about that. Zapier and automation apps were huge before agents, or even for integrations for Slack. There is definitely a big portion of tech products that have mutual benefits by providing good APIs to be in the same bubble

Interesting. It's just an agent loop with access to python exec and web search as standard, BUT with premade, curated, 150 tools like analyze_circular_dichroism_spectra, with very specific params that just execute a hardcoded python function. Also with easy to load databases that conform to the tools' standards.

The argument is that if you just ask claude code to do niche biomed tasks, it will not have the knowledge to do it like that by just searching pubmed and doing RAG on the fly, which is fair, given the current gen of LLM's. It's an interesting approach, they show some generalization on the paper(with well known tidy datasets), but real life data is messier, and the approach here(correct me if im wrong) is to identify the correct tool for a task, and then use the generic python exec tool to shape the data into the acceptable format if needed, try the tool and go again.

It would be useful to use the tools just as a guidance to inform a generic code agent imo, but executing the "verified" hardcoded tools narrows the error scope, as long as you can check your data is shaped correctly, the analysis will be correct. Not sure how much of an advantage this is in the long term for working with proprietary datasets, but it's an interesting direction

>if your code doesn't work it doesn't work

you can't bullshit a computer

this is wrong. I would argue the difference between a junior dev/intern and a senior engineer is that while both can write code that works, the juniors find local maximas, like solutions that work, but can't scale, or wont be very easy to integrate/add features on top/maintain etc.

This happens in maths, biology, in all science fields. Experience is partly the ability to take decisions between options that both work.

This is why coding assistants are amazing at executing things you are clear on what you want to do, but can't help (yet) on big picture tweaks

Haha we are in such a bubble. All engineering concepts are being thrown in the trash. We already have problems with prompt injections in MCP (eg supabase thread yesterday). Imagine the possibilities when you allow the LLM itself to write and execute arbitrary code that directly has access to sensitive info. This is the equivalent of allowing a malware to not only just execute code, but give the malware developer free reign to write his bespoke code for your machine.

Outside of that, writing your own tools on the fly it's clear waste of tokens/money, plus makes benchmarking and evaluation way more difficult, it's another degree of freedom etc. People never think with first principles in mind anymore. Just the quickest cute solution to anything, with not the slightest consideration any side effects

I mean that's the dirty secret of any RAG chatbot. The concept of "grounding" is arbitrary. It doesn't matter if you use embeddings, or use a tool that uses your usual search and gets the top items, like most web search tools or google's. Is still relies on the model to not hallucinate given this info, which is very hard since too much info -> model gets confused, but too little info -> model assumes the info might not be there so useless. The fine balance depends on the user's query, and all approaches like score cutoff for embeddings etc just don't generalize.

This is the same exact problem in coding assistants when they hallucinate functions or cannot find the needed dependencies etc.

There are better and more complex approaches that use multiple agents to summarize different smaller queries and then iteratively buildup etc, internally we and a lot of companies have them, but for external customer queries, way too expensive. You can't spend 30 cents on every query

CT-ART 4.0 is the gold standard. Again, not fully free, but it has some very instructional features, like playing against alternate moves, solving mini versions of a puzzle, playing the opposite side etc. Used it for years.

you misunderstood. its 300ms just for the apply model, the model that takes your coding models output (eg sonnet) and figures out where the code should be changed in the file. Cursor has its own, and claude uses a different technique with strings as well. So its 10sec vs 10sec +300ms using your analogy

Glad to hear quality comes first! Then I assume you have some public benchmarks like the ones you mention that are reproducible? I could only find this graph https://docs.morphllm.com/guides/apply but there is no mention of what it refers to, what data it used etc.