The challenges of producing quality code when using AI-based generalistic models | Hacker News Reader

The challenges of producing quality code when using AI-based generalistic models

infoq.com

12 pointskiyanwang4 comments

This is one of the things that bugs be about AI code generators. They are pretty helpful when I'm writing quick generic routines. But they don't work for anything slightly novel (I write research code). The other big factor is that the models are trained on GitHub, SO, Reddit, etc and have no real quality filter. That's good for breadth, but not for depth and high quality code[0,1]. I'm already a shitty coder, it's not that helpful to have a companion write other shitty code who doesn't understand the whole ecosystem we're building within nor the abstract idea we're trying to build.

I hear a lot about how people say they can't live without copilot now and I'm often confused (not new programmers, but I'm hearing SWEs say this). Is it because we're all just writing junk code as fast as possible and say fuckall to technical debt? Am I using the tools wrong? (Not sure how) Or what is going on? I really just hear these two opinions "omg it changed my life" and "yeah, I get some use out of it, but just saving me a google search mostly." Is my sampling just really biased?

[0] yes, there's lot of high quality code on github, but it is dominated by a lot of shitty code. And high quality code is often high quality because of the complex interconnections between many different routines which is a level of complexity LLMs don't yet have and requires quite large context lengths. We're getting there but there's still good performance degrade as context length increases even if all the text fits within that context.

[1] Don't get me started on HumanEval as a test set...

Is it because we're all just writing junk code as fast as possible and say fuckall to technical debt?

My guess is : probably, I love taking time to make architecture because it's fun and rewarding, but I would not be shocked it's the case, and in fact the right thing to do in a lot of case. 90% of your code will be thrown away anyway in the majority of cases, probably due to external causes. Also and that's probably where copilot shine : for lot of people 90% of your work is adding field/filters functions to a crud page or making a shitty demo for a pet project of your company.

The value is 'when can I show it to a potential client' and not 'is the architecture state of the art ?'.

Of course once the demo is sold, your 3 days prototype with a big "For demonstration purpose only" warning may end in production and you will have 1 afternoon to remove the warning and 'Polish the thing so it's bug free' but hey, that's another problem !

Joke aside, and the book have a lot of very opinionated and contestable content, I remember having read 'Refactoring' by Kent Beck and Martin Fowler and 'forced' myself to work on refactoring, when all my coworkers and myself were prompt to "reboot when possible" all 'old' projects or messy prototypes and I now think exept for rare cases : hacking, testing and getting feedback then refactoring while hacking new features is surprisingly efficient time wise even if you rewrite ten times a routine. When applicable of course, we are not talking medical hardware drivers).

And about your main 'interrogation' I would guess copilot is not specially efficient for you because you are seems to have an academy background and the experience about 'reading the docs' or 'making a google search'. Making even a senior software engineer "read once the documentation of your framework" is an impressive feat in a lot of software companies. Most people seems to prefer the "stack overflow look for/copy/paste/pray workflow" and copilot is basically an evolution of that.

Yeah it does make sense that my coding may just be very different from the average person on HN. I write a lot of code that has "never been seen before" per say. The main things that are generic for me are really just some basic loops for setting up pytorch DDP and calling my train function. But even those get modified. I do have some more boiler plate stuff but every copy to a new project gets more edits and improvements because of changes and learning from previous failure points (e.g. learning that I should enable my custom logger to track another thing because that was actually important for reproducibility).

But then again, I know people who work in similar areas that swear by it and I just can't figure it out. They tend to not share their code though so I'm not sure if it's truly bad or not. These people tend to have a CS background (undergrad) whereas I do not though so idk.

>They are pretty helpful when I'm writing quick generic routines.

That is precisely the point. If I can reliably outsource the done-1000-times-before type tasks to AI which completes those small tasks several times faster than I could, then I see that as a win. It doesn't replace me, not even close. It does free up more of my time for actually taking lunch or cracking more interesting problems though.

Reply on news.ycombinator.com