136 karma · joined November 23, 2024
I haven’t experienced anything like that. My first inclination is to shout: skill issue. Because steering these language models is our responsibility. Reading 10 files doesn’t lead to 10% usage gone even on my $20 subscription. Try talking to your internet friends and find a better way to steer these agents.
I also remembered the sheer joy of forking enterprise codebases like that of cal.com and highlight.io - just to study and try to understand how large teams worked. I followed issues, I read through PRs and related comments, looked at how some of the comments were resolved. I studied the various moving parts of the app - highlight especially being effectively a log ingestion and aggregation platform, had some rather large infrastructure requirements - kafka, redis, postgresql, clickhouse. It was hard to run with infrastructure directly on your computer. You had to use their docker containers.
I also remembered reading through a tutorial for an sqlite clone in C. It was interesting. I particularly enjoyed typing out the C code and even trying alternate implementation of b-tree operations.
I remember back in 2022 - after I was unemployed for a while - I discovered open source bounties on algora.io (LLMs have since made this platform go bust). I was able to earn roughly $300 per month for a while working on open source issues ranging from $50 to $350. Now most $50 bounties can be one-shot with frontier models. So of course, no one posts open source bounties anymore. If you do, you may just end up with 50 slopped PRs.
Just random thoughts.
Your whole scientific endeavor in this seemed to thrive on the rock solid foundation of tweets.
And all those referencing of the Kelly post makes this even more of a clickbait. All you had was to put all the ingredients of a post that will draw the crowd but offer very little on the subject matter it purported to discuss
1.”Writing software was never the hard part” - isn’t saying coding was easy. The comparison was against building a viable product. Doing business involves a whole lot more ambiguity than sitting in your room writing code. You could build anything and yet the hard part was building something that matters
2. >> Nobody knows how the AI revolution will play out in the end
I disagree with this statement as far as Software Engineering is concerned. We kinda know. Everyone uses an agent harness to code this days - the only real difference is in how - and that still sets developers apart but most are using similar tools.
This reads more and more like clickbait. Sorry. I always find posts like this outrageous but I suppose that was the author’s intent. Outrage will get you votes. Taking extreme positions in an argument will get you attention. Kuddos
PS: I still exported the table before deploying this fyi.
Personally, since I only do cosmetic changes to PRs like refactoring them to fit into the current codebase and sprinkling some best practices here and there - I haven’t noticed any decrease in coding velocity.
Often, in formal communication, I find it makes a lot of sense to make it as robotic and to the point as possible because informing my boss about the progress of a project shouldn’t leak any emotions. This is not a romantic or social relationship where emotions need to be expressed.
And I like the idea of setting goals for what I want to convert and having these llms go through the message and let me strip out the fat and ensure there is little room for misinformation and digression.
So yeah, what the author wants probably works more in an informal social setting than in a formal.
LLMs tend to make most people feel like they can write code without understanding the problem at hand. In a lot of cases, they even climb the ladder of ai-suggested designs and end up with what is probably poorly designed but works anyway. That probably fuels their confidence and gets them to continue - I get away with some of these things on most reactive UI frameworks.
When doing systems programming however, it is hard to get away with poorly designed, conceived and executed work. Especially in open source projects where a couple of maintainers have to retain context of the entire project over a long period of time to facilitate their ability to review and make community contributions possible. These people tend to understand the product deeply and tend to also do gate-keeping for the quality of code that is contributed. Without that gate-keeping, open source might just not be sustainable.
Today with all these llm tools, people just get up and feel like they can ai-slop their way to PRs on open source projects. This is a maintenance burden on open source maintainers that I fear will only increase over time.
It is probably time for github to implement a policy of enabling maintainers to ban some users from making a PR?
I think at a fundamental level, I expect we can produce higher quality software under budget. And I really liked how you were clearly thinking about cost benefits especially in your setup. I’ve encountered far too many developers that just want to avoid as much cognitive work as possible. Too many junior and mid devs also are more interested in doing as they are told instead of thinking about the problem for themselves. For the most part, in my part of the world at least, junior and mid-level devs can indeed be replaced by a claude code max subscription of around $200 per month and you’d probably get more done in a week than four such devs that basically end up using an llm to do work that they might not even thoroughly explore.
So in my mind I’ve been thinking a lot about all aspects of the Software Development LifeCycle that could be improved using some llm or sorts.
## Requirements. How can we use llms to not only organize requirements but to strip them down into executable units of work that are sequenced in a way that makes sense. How do we go further to integrate an llm into our software development processes - be it a sprint or whatever. In a lot of green field projects, after designing the core components of the system, we now need to create tasks, group them, sequence them and work out how we go about assigning them and reviewing and updating various boards or issue trackers or whatever. There is a lot of gruntwork involved in this. I’ve seen people use mcps to automatically create tasks in some of these issue trackers based on some pdf of the requirements together with a design document.
## Code Review - I effectively spend 40% of my time reviewing code written by other developers and I mostly fix the issues I consider “minor” - which is about 60% of the time. I could really spend less time reviewing code with the help of an llm code reviewer that simply does a “first pass” to at least give me an idea of where to spend more of my time - like on things that are more nuanced.
## Software Design - This is tricky. Chatbots will probably lie to you if you are not a domain expert. You mostly use them to diagnose your designs and point out potential problems with your design that someone else would’ve seen if they were also domain experts in whatever you were building. We can explore a lot of alternate approaches generated by llms and improve them.
## Bugfixes - This is probably a big win for llms’ because there used to be a platform where I used to be able to get $50s and $30s to fix github bugs - that have now almost entirely been outsourced to llms. For me to have lost revenue in that space was the biggest sign of the usefulness of llms I got in practice. After a typical greenfield project has been worked on for about two months, bugs start creeping in. For apps that were properly architected, I expect these bugs to be fixable by existing patterns throughout the codebase. Be it removing a custom implementation to use a shared utility or other or simply using the design systems colors instead of a custom hardcoded one. In fact for most bugs - llms can probably get you about 50% of the way most of the time.
## Writing actual (PLUMBING) code . This is often not as much of a bottleneck as most would like to think but it helps when developers don’t have to do a lot of the grunt-work involved in creating source files, following conventions in a codebase, creating boilerplates and moving things around. This is an incredible use of llms that is hardly mentioned because it is not that “hot”.
## Testing - In most of the projects we worked on at a consulting firm, writing tests - whether ui or api was never part of the agreement because of the economics of most of our gigs. And the clients never really cared because all they wanted was working software. For a developing firm however, testing can be immense especially when using llms. It can provide guardrails to check when a model is doing something it wasn’t asked to do. And can also be used to create and enforce system boundaries especially in pseudo type systems like Typescript where JavaScript’s escape hatches may be used as a loophole.
## DEVOPS. I remember there was a time we used to manually invalidate cloudfront distributions after deploying our ui build to some e3 bucket. We’ve subsequently added a pipeline stage to invalidate the distribution. But I expect there are lots of grunt devops work that could really be delegated. Of course, this is a very scary use of llms but I daresay - we can find ways to use it safely
## OBSERVABILITY - a lot of observability platforms already have this feature where llms are able to review error logs that are ingested, diagnose the issue, create an issue on github or Jira (or wherever), create a draft PR, review, test it in some container, iterate on a solution X times, notify someone to review and so on and so forth. Some llms on this observability platform also attach a level of priority and dispatch messages to relevant developers or teams. LLms in this loop simply supercharge the whole observability/instrumentation of production applications
But yeah, that is just my two cents. I don’t have any answers yet I just ponder on this every now and then at a keyboard.
Even though it gives me the vibes of something the fictional " Sirius Cybernetics Corporation", would do.
And your subprime mortgage reference - suggesting they are manipulating information to inflate the value of the firm - doesn't cleanly apply here. For once, here is a company that seems to have faithfully represented their obscene losses and here we are already comparing them to the likes of enron. Enron never reported financial data that can be categorized as losses.
I see lots of people speculating about these losses and I really wish someone investing in openai could come out and say something vague about why they are investing.
Once again, I need not tell you, the information available to the general public is not the same as that which is available to anyone that has invested a significant amount into openai.
So once again, reign in your tendencies to draw conclusion from the obscene losses they have reported - especially since I'm positive you do not have the right context to be able to properly evaluate whether these losses make sense or not.
In this case however, what you don't know is more relevant than what you do know. Despite the author's knowledge of publicly available information, I believe there is more the author is not aware of that might sway their arguments. Most firms keep a lot of things under wraps. Sure they are making lots of noise - everyone does.
I'm filing this under click-bait.