I don’t think anyone is disputing that it makes many kinds of work easier and/or faster, just the degree of those improvements and how much of those improvements actually translate to dollars earned or saved.
Seeing hard data on that would help settle the issue.
Nobody even tries to measure this. Their CEOs and board members have AI shares, and potential investors need to see your AI story to justify your valuation. It's a prison of lies.
If developers are shipping more code (and most of the enablement is in software right now) does that show up in top line? Maybe… if sales and marketing are also selling more, and product people are building the right thing, and…
It’s not a straight line from AI -> $$$
That seems like it might be a bit of a problem for those frontier model CEOs who have been making exactly the opposite claims.
Most of the claims I’ve seen are around productivity (bottom line).
And while it’s hard to say “we would’ve had to hire more devs to be this productive without Claude” it kind of feels like the case, doesn’t it?
Or maybe it was! And investors were actually great humanitarians!
Come on, you know better.
With AI, keeping good quality gets even harder because it normally only produces tricky defects.
But I also think:
- there are a lot of companies that don’t need to be 5 9s[1]
- I have a feeling it’s possible to ship fast without drastic quality drops
[1] Claude code is 99.85% available and is one of the world’s most important products
Really? If we delete CC today, will humanity stop functioning? Nope. It's so far from one of the world's most important products.
About 5 9s, reliability is just one of the axes which suffers from fast generation.
There are a lot more: architecture, accuracy, etc...
> I have a feeling it’s possible to ship fast without drastic quality drops
at what scale and for how long?
To call Claude Code not a world defining product is so absurd it’s genuinely laughable.
I guess that’s why they’re a $2T company within like 5 years of founding with or without your support.
Now you’re just being obstinate.
I see this in a very non-AI way working at a company that’s a hodgepodge of acquisitions. Some of our products just don’t make a lot of money, don’t resonate, or are behind the competition in various ways.
The amount of code or general engineering complexity of the application seems completely detached from profitability a lot of times.
Now the bar is going to be higher for software to actually be worth buying when it’s so easy to make your own for yourself, so that also means that our ability to pump out code loses value.
Of course, they're almost entirely unrelated. It's just programmers believe solving difficult problems with specifically code is valuable in of itself. The goal is to get as much value as possible from as little code & complexity as possible.
> I see this in a very non-AI way
That's very non-AI of you
I don’t use AI to write prose at all in any form, not even drafts.
This is FUD. It isn't necessarily a good or bad thing.
At the very least, using AI to audit your code is responsible use. The testing is the same as it ever was. If there was a chronic problem with shipped AI assisted code, it would be a popular talking point.
> If quality isn't controlled properly, it eventually shows up as negative in the top line.
This an idealistic pillar of capitalism that has been falsified across time. Depending on what someone means by "controlled properly", I don't think this wasn't been true before AI and I don't think it's demonstrated post-AI. Momentum (notably capture) keeps the software markets resistant to quality defects.
Think security, optimization work, refactoring, etc. Today everybody is drunk with AI, but what happens when the hype is over and you can’t justify to the finance dept. why the company is spending X on tokens?
Agents accelerated rollout of a data model change by at least 2 months of eng effort.
Those unblocked a $100k/month infra downsizing. That 2 months pays for my team's AI spend for the year.
Well yes when your tokens are nearly free, then getting literally any business value is a good outcome.
That "je ne sais quoi" is called hygiene and it's hard to measure but when you stop cleaning, generally bad things happen.
That's about as hard of data as you can get.
Are the consultants delivering more billable hours per person now?
All sorts of reasons why the answer to that may be: no.
You seem hellbent on disproving me. Why?
Uhm, no. The OP makes specific claims wrt to AI and financial payoff; you offered up a case study, with incomplete data, as possible evidence that AI is generating these financial payoffs. I'm asking questions to uncover the data necessary to reconcile your case study against the actual claims from OP.
> Even if they were FTEs, it would mean the consultants could do ~3x the work.
Could. But again, are they? That's the whole point of Apollo's claims: people keep saying AI can or could increase capital efficiency, but it's not showing up in any actual financial results.
You're extrapolating a 3x increase in throughput from the optimization of a single process that exists in a whole business system. Given your background, I don't think I have to explain to you why that doesn't make sense.
Also, do we think the S&P 493 is so incompetent that they'd miss these optimizations? Or is it more likely that this firm you're referencing is a small inefficient player, and the S&P 493 already realized these efficiencies decades prior without AI?
Really? Not any results?
You're confusing the original article and the parent comment I replied to. I think the Apollo article is way off, BUT I have very intimate knowledge with very concrete numbers of a use case where AI has a very clear and definable ROI.
> Could. But again, are they?
YES. That's what I'm telling you. They just implemented the system. The contractors spend less time on each project which means they have more time to do another project. How is that so hard to understand?
> You're extrapolating a 3x increase in throughput from the optimization of a single process that exists in a whole business system.
Are you deliberately being obtuse? The company very clearly drives ~$5k additional profit per project. They do hundreds of similar projects.
> Also, do we think the S&P 493 is so incompetent that they'd miss these optimizations? Or is it more likely that this firm you're referencing is a small inefficient player, and the S&P 493 already realized these efficiencies decades prior without AI?
I never made any of these claims, why are you saying I did?
For non-technical clients that may well be the case for a while, but I suspect even the most non-technical clients will eventually realize they can claw back a good portion of the efficiency as discounts.
I'm negotiating some contracts where I could do the work in-house for about 1/20th of the previously-reasonable vendor cost. I'd rather not do it, so I'm happy to pay 1/20th plus a reasonable percentage above that, but that's the limit. The vendor can accept that, or goodbye.
Interesting times on vendor contract negotiations these days.
If the outcome is the same then the fees shouldn't matter to the client in the long run. Said differently, if the client who pays the management consulting fee gets the same return out of the service, why do they care?
Now, what may end up happening is that some other management consultant vendor in the market comes in an says "ya know what we're willing to do the same work for less" (which means their margin goes down). This could indeed drag their pricing down.
> Interesting times on vendor contract negotiations these days.
For sure.
At least at every company where I've been in charge of my vendor contracts, I am always grilled by management on how we can lower the cost of the contract and whether we are getting maximum efficiency from each vendor. There is huge pressure every year to find ways to lower the cost.
The teams throughly reviewed and brought up 2 overlooked cases, but mostly we all reminisced about things forgotten that AI discovery had turned up which would have been totally overlooked otherwise. Forgotten discussions, designs, outages, odd decisions, weird edge cases prepared for but never encountered, etc. Then they were migrated, it went smoothly, all to save a low seven-figures per year dollar amount.
The fundamental approach of exhaustively researching to that degree would never have been possible without AI. It's just not feasible to analyze 5 years of even one of those discussions sources (e.g. slack) to figure that out, let alone ALL of them. Now a team of ~1.5 engineers has tackled all that by leveraging AI.
The hard data is piling up in companies, but it'll take a while to get out there.
(Numbers and details lightly greeked in the above account)
This is the first cycle in my (relatively) long career where the demand for return felt instant. The tools and techniques are almost comically immature and the average person interfacing with the tools is largely bewildered.
For word processing for instance people spent years upon years coming up to speed on wordstar, word perfect, word. Decades later they’re a common part of life and the tangible benefit over secretarial pools is obvious - there is no such thing as a secretarial pool.
AI it feels like almost a blink of an eye and the demand is “well, it’s been a few years, where’a the post scarcity society I was promised?”
Next step is 9to9/7days for the American worker to compete, or else you'll be left behind
A new technology can deliver significant increases in real economic value in terms of utility per $ expended without significantly increasing revenue for the average firm.
As an example, my X is that boilerplate and general speed/effort of modifying my code is lower, but I’m noticing that this doesn’t necessarily make my team any more effective overall.
It seems like it’s been a code review burden and it’s led to some team members being too ambitious with picking up more projects that don’t make sense.
I have seen some managers in particular toss prototype work that they shouldn’t be doing at underlings, and one of those projects in particular really ended up taking me longer than if I had done it from scratch without AI. The fastest result would probably been if I did it from scratch with AI, but because I took it over from someone who didn’t understand the fundamentals and farmed out decisions to AI I had to spend a lot of time correcting mistakes.
Obviously, good management beats bad management all day long, but I bet you most companies don’t have stellar management.
Our company has secured projects and executed them to a standard which we would not have otherwise were it not for AI or hiring more people. Now, these are at least 3 year projects, so we are only seeing ~3 being AI accelerated.
For us, yes, it's a multiplier, directly translated to more $$.
At what stage did the engineering fail?
In a webpage the UI design, authentication, etc. are all “boilerplate” if one just wants to do “X” instead of worry about infrastructure.
I see dozens of teams that are doing cloud to the wazoo when really they need to just be told “we’ll load your routes from main.py, build your assets in /static, you need a DB click the button to provision it”.
It boggles the mind that so many teams are reinventing the wheel a million times over to publish their APIs, register AWS accounts, setup CI, decide on ECS vs Lamda, etc.
That’s pretty much the answer. The simple stuff has already been tried and there’s already some reason why we can’t just make XYZ simpler.
I look at stuff like terraform code or AWS infrastructure and it’s annoying and repetitive as hell sometimes but at the same time it’s incredibly impressive that you can just describe something as complex as your own private company’s network existing on global data centers and on a relative level you can just bake in pretty complex stuff like a spiderweb of IAM rules, disaster recovery/backup/databases you barely have to think about, horizontal scaling, security policy and all kinds of other stuff I’m forgetting off the top of my head.
It’s heavy boilerplate because like you said you can basically design an infinitely complex corporation as code and once you’re done with that deploy it instantly which used to be pretty much unheard of.
The AI analyses sample data from old and new system, and then builds a prototype deterministic mapping pipeline. The experts can then inspect said pipeline and with their domain expertise catch edge cases/exceptions/uncovered cases, fix them and improve things, and then you use the vetted deterministic pipeline for migration. Yet, the initial AI draft saves enormous time.
But the problem identified that is talking to people and having them record data.
The stack certainly could need improvement, but much of the overhead comes from the process being pretty serial because each step depends on the previous. If someone we needs to talk to goes on vacation, sick or has a full calendar this delays it much further. Iteration speed is not linear in such cases. We do a lot of custom solutions and have to implement custom parsers or interface with undocumented ancient software or simply no existing software. Everything is custom, because previously they have no processes, no IT infrastructure or simply only data intended for humans (roughly structured, informally written, ever changing naming schema, typos).
This area is one which AI has shown to be pretty good (as long as some error rate is acceptable). Character recognition, voice transcribing, converting documents from one standard to an other. I can see an argument that if people spent most time doing those things, AI will speed it up (with a bit of added errors).
I am reminded however that in translations and AI generated subtitles, there is still a lot to be desired compared to the human touch. Handwritten papers in a different language, using local short hand and assumptions, may result in some odd and error prone AI transcription.
When you look at old-school code that just hacked things together, it was ugly, but also very easy to in turn further hack. There was no elegance or grand base types with 80 levels of inheritance to enable clean extension at every possible juncture of types. And no spending forever to find out the exact function you want to be able to override, only to discover that it's outside your modifiable codebase and wasn't marked virtual. Everything just worked.
What I've seen (at least at one place I worked) is that the code may be very well engineered for further modification - but only by people who have worked quite a bit on that code. For someone coming from outside, it's almost impossible. It takes forever to find your way around, and figure out where the magic place is that you need to modify in order to work within the existing design.
Well, you say, that's what the design documents are for! But they didn't have enough detail. They described the major objects, but not all the ancillary helper objects. They didn't describe the flow of data through the system in enough detail to know what to modify. The knowledge was there, in peoples' heads, but it was hard to get those peoples' time.
So, yeah. "Well engineered", but not engineered for maintenance by people who are not those already in the group.
That's like saying "that's the irony of it, a well engineered bridge sometimes drops chunks of concrete onto cars that pass underneath".
Engineering isn't about creating some perfect thing that is prepared for any conceivable scenario, but about solving a problem as efficiently as possible.
speed of iteration is more important than quality of iteration; high iteration speed lowers the cost of failure and compounds improvements
Moving the goalpost is not compelling.
I also find that thinking through a design first helps me achieve a result faster. So I prefer languages that encourage you to think first, type later. Interpreted languages encourage you to type first, think later.
AI is going to effectively let random people compete and displace big tech companies with free software they simply release. This is why google wants to lock android down, they see it coming. CEO led companies with layers of management and lots of investors are obsolete. We are going to go back to teams of 1-6 people doing all the work on their own like in the 90s.
I understand there's some benefit of this. But is it worth putting the global economy into a death spiral? So ManBeardPc can save 4-9 business days to go and say "yeah, that doesn't work".
That’s the whole distinction
I think it sums up to something along the lines of "you get what you pay for".
So, AI is making 5-10% of your project much faster. Whoopdie doo.
The firms of 2031 won’t be doing business like the firms of 2021 but just using AI for coding or in a few places to accelerate the same workflows.