I don't think anyone at our shop has high hopes for LLM's in programming beyond efficiency anymore. I'd like to see Github copilot head in a direction where it's capable of auto-updating documentation such as JSdoc when functionality changes, LLM's are already excellent at writing documentation on "good" code, but the real trick is keeping it up-to-date as things change. I know this is also a change-management issue, but in the world where I mainly work, the time to properly maintain things isn't always prioritized by the business at large. Which obviously costs the business down the line, often grievously so, but as long as "IT" has very little pull in many organisations it's also just the state of things. I'd personally love for them to get better at writing and updating tests, but so far we've been far less successful with it than the author has.
As far as efficiency and quality goes our in-house measurements point in two directions. For inexperienced developers quality has dropped with the use of LLM's, which in our house is completely down to how employees (and this is not just developers) tend trust LLM's more than they would trust search results. So much so that a lot of AI usage has basically been banned from the wider organisation by the upper decision makers because quality is important in what we do. Yes I know this is ironic when you look at how they prioritize IT in an organisation where 90% of our employees use a computer 100% of their working hours. Anyway, as far as efficiency goes there are two sides. When used as fancy auto-complete we see an increase in work output across every kind of developer, however, when used as a "sparring partner" we see a significant decrease. We don't have the resources to do a lot of pair-programming and a couple of developers might do direct sparring on computation challenges for 1-2 hours a week. They are free to do so more, and they aren't punished for it as we don't do any sort of hourly registration on work, but 1-2 hours is where it's at on average. Sometimes it'll increase if they are dealing with complex business processes or if we're on-boarding some one new.
> Copilot is very useful to scan existing code for any errors or missed edge cases
Aside from tests I think this is the one part of the article I really haven't seen in our very anecdotal testing. But maybe this is down to us still learning how to adopt it properly or a difference in coding style? Anyway, almost all of our errors aren't with the actual code but rather with a misrepresentation/misunderstanding/unreported-change-in of business logic, and this has been the area where LLM's have been the weakest for us.