Yeah surprisingly the text of this blog is almost entirely generated by AI :) I gave all the ideas though
77 karma · joined June 18, 2025
We dug into 17 billion tokens of behavioral data across 413K AI agent trajectories (CoderForge-Preview) attempting real GitHub issues. Instead of just looking at final SWE-bench scores, we compared successful runs against failing runs on the exact same problem to filter out task-difficulty confounds.
The biggest surprise? Agents are not junior developers, and prompting them to act like humans actively hurts their success rate.
Here is what the data actually shows:
Human exploration rituals predict failure: "View-before-edit" and "grep-before-edit" are negatively correlated with success. Humans do this to build mental models. Agents already have the codebase in their context window; if they are heavily grepping, they aren't learning, they're flailing.
TDD is the ultimate predictor of success: The single strongest behavioral signal of a passing agent is the fraction of early bash commands dedicated exclusively to running the test suite.
The Single Responsibility Principle is law: Agents that scatter edits across 3 or more files in the first 30% of their run see their success rate plummet. Successful agents fix one targeted thing at a time.
Perseverance is a myth: If an agent runs the exact same bash command twice early on, it’s a massive failure signal. They don't adapt; they just get stuck in a loop.
Check out the article for full content!
The bigger issue is benchmark saturation. We highlight "circle packing" in the post as a prime example: four different recent papers (ThetaEvolve, TTT-Discover, AdaEvolve, and EvoX) all converge to the exact same ceiling (2.635983). Once multiple methods collapse to the same endpoint because they are essentially just wrapping the same SLSQP Python script, the benchmark stops measuring open-ended improvement.
To fix the evaluation gap, we built Frontier-CS. It consists of 172 algorithmic tasks and 68 research tasks, designed by ICPC World Finalists and CS PhDs.
Our main goal was to create environments with massive search spaces that require actual information-efficient exploration and long-horizon reasoning (like our hidden Permutation task), while still keeping the scoring 100% deterministic so we don't have to rely on "LLM-as-a-judge." SkyDiscover is already using this for 200+ task benchmark suites to get much cleaner, less noisy data on agent performance.
A while back, we shared our open-source project LMCache here and were blown away by the incredible support and feedback. Today, our team is thrilled to share more about one of our core components: CacheBlend. Recognized with a Best Paper Award at ACM EuroSys 2025, this technique is a pain killer for efficient RAG applications The Problem: Your KV Cache is Wasting Potential In modern LLM applications like RAG and Agents, we constantly feed the model new context. For example, in RAG, we retrieve relevant documents and stuff them into the prompt.
The issue is that this dynamically retrieved context doesn't always appear at the beginning of the input sequence. Traditional KV caching only reuses a "common prefix," so if the new information isn't at the very start, the cache hit rate plummets, and your GPU ends up recomputing the same things over and over. The Solution: CacheBlend - 100% Hit Rate, No Compromises CacheBlend changes the game by allowing for the reuse of pre-computed KV caches regardless of their position in the input sequence.
This means we can finally achieve a 100% KV Cache hit rate in applications like RAG. The performance gains are significant:
Faster Time-To-First-Token (TTFT): Get your initial response much quicker.
More Throughput: Serve significantly more users with the same hardware.
Almost lossless Output Quality: All of this is achieved with little degradation in the model's generation quality.
How does it work? CacheBlend intelligently handles the two main challenges of reusing non-prefix caches: Positional Encoding Update: It efficiently updates positional encodings to ensure the model always knows the correct position of each token, even when we're stitching together cached and new data.
Selective Attention Recalculation: Instead of recomputing everything, it strategically recalculates only the minimal cross-attention needed between the new and cached chunks to maintain perfect generation quality.
For detailed analysis, please refer to the official paper: https://dl.acm.org/doi/10.1145/3689031.3696098
Where can I try it? Our official repo is at: https://github.com/LMCache/LMCache The newest interactive CacheBlend demo at: https://github.com/LMCache/LMCache-Examples/tree/main/demo-r...
Ask us anything!
In LLM serving, the input is computed into intermediate states called KV cache to further provide answers. These data are relatively large (~1-2GB for long context) and are often evicted when GPU memory is not enough. In these cases, when users ask a follow up question, the software needs to recompute for the same KV Cache. LMCache is designed to combat that by efficiently offloading and loading these KV cache to and from DRAM and disk.
Ask us anything!