springtimesun

My partner spent many years in big law and now sits across the table on the client side. The billable hour has been problematic for much longer than AI has been in the picture, but what many people forget (including sometimes the companies paying the bill) is that you often aren't really paying for the hours, you're paying the structural tax that has been established by the system. Hours imply work product, but sometimes the reason firms are paying outside counsel _at all_ isn't because their in-house team couldn't do it, it's because the market/courts have told them they need the liability transference granted by a signed piece of paper.

That said, the rest of the article is on point. AI billing analysis software + internal knowledge of how the sausage gets made has shaved literal millions off bills at this point. I wouldn't expect them to lose money in an absolute sense though. More like a balloon you squeeze from one end.

One thing I came here to say, but turn out not to be the case, was the large jump in associates salaries as an interesting possible driver. I found a handy tracking number that does inflation adjustment. Turns out when you index for inflation, the associate pay scales are more or less flat.

This is a question I have been circling around. The agents are good at Ruby/Rails. It’s probably a confluence of things, but whether the L in LLM benefits from Ruby’s emphasis on human readability is something I’d love to see a study on. Or more generally, how key is (human) language itself in the performance ceiling of LLMs. A machine specific coding language/framework/process could obviously be more efficient, but if language itself is the hard limit, those would be dead in the water with LLMs. Or put another way: if language wasn’t the ceiling, why didn’t we see machine first coding implementations before LLMs?

This matches my experience. My solo projects are now bigger than I ever would have accepted alone. The one I’m going to finish soon is one I actually refused a year ago because it was too big.

It’s the top of the chain that I wonder about. Agentic development is so fast (and in the process I’ve developed so good) that I’m thinking there is actually a maximum size a company doing this work could get. Beyond that top end size the management overhead is what’s limiting or the market size total. Maybe not, maybe mega dev houses will find other problems to solve, but I have a hard time imagining it.

I have this in a rules file. It respects it. The only thing that broke recently was when they added the session links. That needed an explicit additional rule. Sometimes rules don’t get added to worktrees but I haven’t had the patience to isolate why this only happens sometimes.

This is where I landed when a recent(ish) change broke symlinking for rules files. I didn’t like how hacky it was a first, but now I’m glad I did it. The same hook drives specific rules the non-Claudes get per repo.

I was in the same boat around March. I started with Claude and giving it really specific instructions about what I wanted. That turned in to more interactive planning sessions where after we nailed everything down I said, ok just do it and I reviewed. The more I used it the more I let go of some things and the more I held on others. Now I have adapted my entire approach to use both local and cloud agents. Start small. Find what fits you. Also try out different models. Closed source is very strong, but you may actually prefer a bit less capable model that you give more direction to.

Yes, I'm with you 100% on the setup. You can go as deep as you want down the hole. At some point I had ask myself what I was really after with the benchmarks and where the line of diminishing returns was (and more importantly, was I really getting generalizable results).

I'm currently using a 2 tier grading system, one pass mechanical (LLM graded) one pass human. The grading ranges from -1 to 0 for the mechanical pass and -1 to 1 for the human pass. A model that does everything competently passed with a score of 0. Non-pass gets a -1. For the mechanical pass that's it. For the human pass there's a chance to get an extra point for exceptional work. Accepting the subjectivity in the human pass was part of the "what am I doing this for" point from above.

The scores then don't get read left-to-right and summed, but rather vertically per task and category. The temp setting as another commenter noted means you can't say anything definitive about the quality from a single run. What I think you can tell is 1) per task, who is the outlier in either direction and investigate, 2) how a task is generally handled by the models and 3) in the aggregate, who is dropping and gaining points consistently and does that change per task type. In the end, it's high touch. It doesn't give you one number to cut or keep a model. It tells you where to look and where not to.

You're the second person running an evaluation business to express interest, maybe I should start one as well ;) I will be reading your blog posts to see if there's anything I can adopt. If I was to give you the objection I would pose if this was an MBA brainstorming sessions I would ask:

Who are you targeting? and if the answer is engineers, what's to stop them from using the LLM to do it themselves? The problem space exists in the easy/hard domain; easy to set up, hard to refine because of nuance and specifics. Moreover, an engineer whose job might be under pressure because of LLMs would be highly motivated and highly capable to create a testing and evaluation setup and "own" this part of the process since it can't be replaced by an LLM.

Personally, I think this space is good consulting territory. There is hard-won knowledge that is valuable and sale-able, but beyond the basic setup, everything has to be tailored. It's also going to be hard to get people into a product moat. Git and IDEs give customers (if they are engineers) all the tools they need for this work and they already know how to use them.

One last note of your comment about same model agents. This is definitely something I have noticed. It's probably structural, but models absolutely prefer their own outputs to others. You can also move them toward agreement or disagreement by telling them that their model (in another session) produced some output. Kimi in particular will dunk on other models and promote what it thinks is its own output. Kimi is also highly sensitive to being bench marked. I have a specific audit script that goes through thinking traces to see if the model realizes it's being tested. Kimi figures it out more than any other model and will dig big time to find other results if you don't completely isolate it. Claude gets pissy if it thinks you are trying to benchmark it. Fable has inserted several times "not under testing" and "not a competitor" when left to its own committing. Fable is also very good at setting up open weights models and testing them. They must have spent a lot of time going for thoroughness and scientific method in training.

Yes at temperature 1 it would need to be averaged. The mechanical scoring is hidden specs through and if they don’t pass that it’s not often a result of temp, but instruction following or verification (different flavor of instruction following given what they are told to do). What’s surprising is just how well they do on this dimension even with the variance. On my tasks anyway, maybe Rails work isn’t that complicated. It’s probably not that I am an amazing ticket writer.

I have run multiple with the same model through issues to see the output. What differs run to run on the same model is style, likelihood to find subtle bugs and chance to correctly guess author intent. I don’t score them on those things though, only note them.

To me, multiple vertical samples is more useful to the coder who isn’t benchmarking these as their job. The average is abstract to you if you’re not going to pointing them at a ticket more than once, but seeing their output over 10 different tickets will give you a window into model tendencies, strengths and weaknesses and help you understand how to write for them. Overall the exercise has made me a better issue writer.

Curiosity and anxiety.

I rebuilt my entire workflow around agents so the unease that the frontiers would change something (access, pricing, availability) and lock me out of that were high. Also why I spent way too much on hardware (at least that can be deducted). Now the whole stack could run in my house and I feel much better about the situation.

Once I got the testing going though it is worth it for its own pursuit. Building processes around the dev process and trying to get the best outcomes is at least as fun to me as actually delivering client code. For the first time in my tech career I feel like I’m in a place with no maps. No one has done my experiments yet. I have a custom quant of K3 at Q5 that lets me get 10 tok/s on a CPU inference box (admittedly you need a 72GB Blackwell also). As far as I can tell no one else has done this. It’s such an exciting time!

In open weights models on a similar metric I find this to be highly specific to role and model. Some models are just diggers by nature, but all will verify more if you tell them to. Have you tried running a Claude watcher with the same role as Sol to see if it verifies as much?

I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR.

The thing that takes the most time is finding the examples. In my real dev flow it’s rarely ticket -> PR -> merge, things bounce around a lot more. So, even though the stated goal is to get away from one shots, that is basically the environment you have to set or else test for specific other outcomes (e.g. agent stopped and raised a question when it realized x).

It takes time to do, but I would really recommend it. Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space.

My takeaway from my testing: in Rails or front end codebases, most models I test are competent and with a human in the loop they would accomplish their goal of getting to a mergeable PR. They are not as good as Claude and since I pay subsidized rates via subscription Claude still gets first pass. They are very worthwhile to layer in as reviewers and catch many issues. My anxiety about a rug pull by the frontiers has been turned way down. I would have to adapt to a local only flow, but it wouldn’t be much adaptation and the opens can deliver in their current state.

Haha, I thought about this today. I'm traveling and walked through a university campus. The new students are coming in and I thought wow, wouldn't it be wild to be a fresh in student in the age of AI (it's been a long time since I was college). But, then I thought, actually this could be an incredible time to be a student. If your curious and motivated the agents can be amazing partners. I think the students who crack how to use AI now are going to be the ones who really run the table as this economic wave crests. But, yeah, you're right on with the Wall-E comment. I would guess many (most?) will turn their brain off and then what's the point of even being there?

There has to be a changing economic undertone to it as well. For a lot of years, the craft that many people enjoyed also happened to pay well and had significant barriers to entry. It's easy to like the craft when it pays you well.

The though experiment I would propose to programmers who only want to write the code is, would you do it if it was paid and regarded like an artist is? You pursue it for its own purpose knowing there is almost no shot of ever being paid or paid well and only a few people may ever read or run it. If the answer is no to that then I would propose that you are actually in it for something other than the craft and its worth settling what that is for yourself.

* I know I am stereotyping artists and there are many kinds and many ways to make your way in the world, it's just a thought experiment.

I thought the last round was the (JS) framework wars

That’s fascinating and gives me even more respect for how well maintainers know their code sections.

I think I am on your arc as well. My learning on different topics is growing every day, but there’s a limit to how much I can absorb. With the LLMs the experiments stay just beyond that horizon and I keep chasing.

Stated too strongly, but I think this could be the model for education (some subjects anyway). Everything personalized to your learning goals, grounded in experiments that give a tight feedback loop and with a model that never gets tired of re-explaining something for the 10th time.

I realized the same while reading it, but my impression is that the output has been well edited. The typical Claude density has been pulled out into something more human readable, but with many of the Claud-isms still intact.

Personally I find this a productive way to produce documents. Claude makes the first pass and I edit line by line. It’s a good understanding check. If I can’t reword something into my own voice it means I don’t understand it. Then have Claude loop again to fact check the rewrites and repeat until clear (or I disagree with Claude on all it’s remaining points).

That’s the same edit loop I used without LLMs, but I get speed in drafting up front (and fact/number pulling) and the check passes make the edits better.

But, Kimi thought for 36k tokens before writing it.

I have been using a PR branch to run GLM 5.3 flash locally so I have been watching the dueling PRs develop to implement it and all the comments and reviews. You're right the scale of it is huge. The speed with which maintainers and community are responding is also super impressive.

I know there's a lot of discussion about AI overwhelming OS maintainers and I can see that. It does seem like llama.cpp is successfully riding that dragon right now though.

I am.

Yes, I thought when I was starting that 1u and 2u form factors were to save space. Maybe they are, but they also have the advantage of moving air front to back very effectively through and over the components. Though there still must be some need because I see even those boxes have optional manufacturer built memory shrouds to try to force airflow between the DIMMs.

I had some 120x38mm fans from another server box that I pulled out because they were too loud and I didn't need the static pressure they were giving. They went in here. That 13mm (and the extra 1k rpm) moves so much more air.

I find 10 to be very usable. It’s not (that) interactive but it chews through tasks. I let Kimi churn away at 4 overnight and it gives good results that are ready for me in the morning.

I’ll be very curious what you get with DDR4. I also almost went that way. I have an Epyc DDR 5 rig and the best I see is 10 tok/s. Caveat being that’s at Q8 and a 4090 doing pre fill so it could be pushed up.

The surprising thing for me is how much work you will need to cool the banks if you’re near your memory ceiling. My memory starts soft throttling at about 74C (dies may be hotter, that’s the bank temp) and will turn down speed to try to stay below 80.

Happy to send my llama.cpp config settings if you want it.

I don't disagree in principle, but there's got to be a ceiling at some complexity. Teach them to prompt their way into a Twitter clone in a month (or less), sure. Teach them to prompt their way into a Bun rewrite in Rust in a month...well, there are apparently some people who need your help right now ;)

This is something devs don’t give enough credit. You may think you’re just promoting, but ask someone outside the field to prompt their way to a feature and you will see very clearly what your experience is bringing to the table.

I’m m just starting to get results on a local/cloud agent pipeline I’ve been building. I don’t think I will ever code again. Plan, review, read, architect, test, yes. Code though simply doesn’t make sense when I’m getting these results at this velocity.

Every few months it feels like I have moved further up the chain. Now I don’t even look at a PR until it’s been coded and gone through at least 2 AI reviews (always different families). Code is truly cheap. I can have the pipeline produce an entire feature as a thought experiment just to see if the shape is the right fit for the codebase. People like to throw the slop term around a lot. It feels like the opposite of slop through. I can have things written and rewritten as many times as it takes until I’m happy with them.

The weakness I see in this is there at the end. The agents never want to throw things away. They just want to keep adding and adding. I think this is my real value in the chain. That intuition built up over years that says this isn’t right yet or that little tingle in your brain that you’ve laid a landline that you are going to step on in 6 months.

Current Rails and Co are miles away from where they used to be then, but even then my feeling was that maybe it was the skill/culture/dev ops at Twitter that was at least partly responsible for the fail whale. But I don’t have any insight as to what was actually going on inside before and after that transition.

I like this approach and I would be pretty happy if this is the way models go, but my question is: go look it up where? Search quality is declining for multiple reasons and I don’t see it becoming better. At best it might be able to outrun the slop, but that means standing still relative to quality now. If we are talking about technical facts or something that can be looked up in docs, sure, but factual questions at search time just defer the (hopefully) considered choice of what to include in weights with the ad hoc decision of what to include from the search results. At least if it’s in the weights then model has a prior to weight the evaluation of sorting through the results.

Maybe that’s an overly negative take. Maybe the search engines will be able to find quality. Maybe humans will continue to write quality. Maybe the models can reason their way in to quality from first principles.

My observation is that finance and banking and corporate functions high enough up the ladder operate more like a clubhouse than cutthroat businesses. You’d think all you need to do is make a better interface, but if you’re not part of the club, you’re not doing the business (or even getting shot at it).

Very clever. You might have said "browser engine" to be more precise and avoid the child comments misunderstanding, but Apple deserves this burn.