40% of the code GitHub Copilot users check-in is AI generated and unmodified | Hacker News Reader

40% of the code GitHub Copilot users check-in is AI generated and unmodified

microsoft.com

156 pointsmfiguiere124 comments

Aside: am I the only one here that is still writing code by hand and not using AI to autogenerate it?

Because it seems over the past couple years on HN everybody is making great use of Copilot and constantly talking about it, while honestly scaffolding boilerplate is the least of my problems, while thinking, designing and abstracting novel ideas into code is the hard part where AI would fall short and just waste my time.

I still do not get if I'm behind the times or just experienced enough (I've been programming for two decades) that I don't feel the need to use Copilot. It's a bit disconcerting because I feel I need to get out of pure programming work ASAP as everybody else is competing with me with Copilot and GPT and I am feeling an existential risk to my craft and livelihood.

Copilot is way more than scaffolding boilerplate, it catches bugs as you are writing them.

For example you expect: x[i]

But copilot suggests: x[i+1]

…and you think, and you think, and then one of two things happens. Either you satisfy yourself that the answer is indeed x[i], or you accept x[i+1] and sheepishly thank Copilot for catching a bug today.

You could have copilot on in the background and never accept a suggestion, and it’s still a productivity enhancement.

I found during my preview months ago that it would often write bugs that took me longer to correct than if I had written it myself. Has it gotten better?

Also in the situation where it suggests x[i+1] and it's wrong, doesn't that annoy you?

In both cases I'm sure this is just temporary while LLMs get better. But my experience was they were _just_ a little too annoying last time I used them.

> Has it gotten better?

yes

Indeed, it introduces subtle bugs related to indexing pretty often. At one point I had it autocomplete maybe 15 lines of code, and I trusted it because the code was all pretty easy and I had written a descriptive comment, which tends to help out Copilot a lot.

However, it introduced a very subtle bug where it used a loop counter `i` where it should have been indexing into an array like `arr[i]`, but it still typechecked and happened not to cause insane errors. Took a lot of time to track down later, and it's a bug I never would have written in myself.

Indeed you are not. I've never used any generative AI tools when coding or otherwise, and continue to refuse to at this point in time.

>I've never used any generative AI tools when coding or otherwise

I'm in the same boat but I must admit the temptation is increasing.

I understand not doing it, just because you didn't get around to try it, but, refusing? why? Moral reasons? Copyright worries?

It's just another tool. So I guess the good old "will never stop using emacs" kind of attitude applies here too?

I never relied on code-completion (not even copilot). I found that coworkers who do simply can't function without it. They simply find it very difficult or even impossible to remember more than a few API names, parameters, etc. I still remember the constant complaints in the early days of Rust when code completion was not available or would constantly break.

Relying on code-completion for coding or relying on google maps for navigation dims your ability to do these things. Given that copilot is a remote service that I do not control, I will not voluntarily tie my ability to write software to it.

Coilot is a conditional, unreliable, untrustworthy, ability-as-a-service. No thank you.

> thinking, designing and abstracting novel ideas into code is the hard part where AI would fall short and just waste my time

How can you be sure of that?

I know scientists who use ChatGPT to rewrite their text to be clearer. They've also used it to pose novel interesting questions about a situation.

On my side, I've found ChatGPT very helpful to explore/compare which library to use to solve my problem and show examples how that library is used.

It's also been very helpful lining up Haskell types and translating GHC error messages into English. Sure, those are non-creative tasks, but it's surprisingly good at them.

Can you trust it though?

It can’t really reason - it’s just very good at guessing the right answer if there are a lot of examples.

It also lies somewhat frequently.

If we have to double check its output, it kind of defeats the purpose - at least part of it - as the “cognitive burden” is still on us.

P.S. Not sure what you mean by “lining up” Haskell types … but don’t LSPs already give us clear information about the inferred types and do so with nigh zero chance of errors.

> Can you trust it though?

Yes of course.

Libraries: I look at the libraries it suggests, see if they have documentation or GitHub activity.

Lining up types: say you have a `List (Maybe a)` and require a `Maybe (List a)`. This is a simple example, of course real life is much more complicated. ChatGPT tells you. Can you trust it? Well did it make the compile error go away?

(Sure, I can think of three ways to turn a `List (Maybe a)` into `Maybe (List a)`. Only one will be what I want, but it's easy to tell whether ChatGPT gave me the one I wanted.)

No, you can't trust it. It will certainly write bugs, and the code will have other issues. Usually too subtle for you to see at first glance, like off-by-one errors, or uncovered edge cases.

I guess if you use it like Google for small nuggets of information you can easily fact check, it's fine.

Although might as well just use Google directly which will send you either to Stack Overflow or Reddit where there are comments discussing the issue with back and forth fleshing it out or to a bunch of tutorial sites where you can source opinions from each site.

edit: nevermind the attempt at your puzzle, my solution doesn't work.

My experience has been that generative AI gives you a useful layer of abstraction above direct search. It’s like having an intern Google stuff for you, check out some promising results, and send you an email summarizing what they found. They make mistakes, but are generally ok.

You wouldn’t actually have an intern do this because the delegation takes more time than doing it yourself. With GenerativeAI, it’s snappy.

Adds another thing that can go wrong though. It doesn't cite its sources either, does it? I have seen people ask it to explain but it's explanation can be wrong too ...

> Not sure what you mean by “lining up” Haskell types … but don’t LSPs already give us clear information about the inferred types and do so with nigh zero chance of errors.

I don't mean type inference but turning the present type a into the required type b. LSP don't do that.

Still learning Haskell, don't quite get what you are saying. But I will probably figure it out someday.

> while thinking, designing and abstracting novel ideas into code is the hard part where AI would fall short and just waste my time.

So don't use it there :) But I'm curious what kind of coding you do that you never have to do any scaffolding, refactoring, or come across any element of repetitiveness.

That is where it shines imo. I find during those boring tasks it quite often just gets my intent, and I can say "yes actually that's exactly what I wanted".

Which allows me to get back to the business of "thinking, designing and abstracting novel ideas" quicker.

AI is not as foolproof as many want us to believe. In my mind it's like giving a less experienced and error-prone engineer the code I'm working on to refactor.

Perhaps it's because I've always been a lone wolf engineer, but I'd rather do it myself than giving quite an important task to someone less experienced than me, whether I'm writing something new (where I need to be at my best) or refactoring (where I still need to be sharp to understand why it was done a certain way to know how to improve)

AI would help with extracting code into a function, or writing yet another algorithm implementation, or wiring bog standard libraries (say, cURL), but that's like foreplay and lower effort work I enjoy doing as a break from the hard stuff.

And to clarify my comment about AI being an engineer with less experience than me, I'd rather be paired with an actual junior engineer, so at least I can mentor them and help them grow, than just having a machine do something that in some cases might be utterly inane, so I still have to waste time double checking its work, but totally powerless to teach and improve it.

It's not foolproof but it does guess a lot of what you had exactly in your mind, just hadn't typed down yet.

If I start "enum { North", I will gladly accept the ", East, South, West }" autocomplete from Copilot, and this kind of thing, in aggregate, saves hours of your time.

This is my feeling exactly, it is a semantic auto-complete. Should/can it write your core business logic? Probably not, but there is a good chance it can infer and suggest logical patterns in implementation.

Again, you’ve basically foregone empiricism and think that the Internet needs your uninformed reckonings about the shortcoming of a technology that you seemingly haven’t had experience with.

It sounds like you’ve been at this for a while, are set in your ways, and are trying to justify it. We all get set in our ways, but stop trying to pass this off as something it’s not.

I’d respect using Copilot and deciding that it didn’t provide value, but all this umming and ahhing is not that.

> In my mind it's like giving a less experienced and error-prone engineer the code I'm working on to refactor.

You have almost 20 years of professional experience on your resume and likely were coding before that; it is difficult for me to believe sometimes, but most programmers have nowhere near that level of experience as the field is constantly increasing in size and scope: when I was in college, Computer Science was one small major people could enter that had only momentarily become popular due to the Internet... now it makes up a third of all students at major institutions that I would have assumed should be doing more interesting research work :(.

Regardless, you (and I) look at this AI as an engineer that is much less experienced than we are, and which is mostly able to code in repetitive ways with a ton of boilerplate that it sometimes gets wrong anyway; but, I think, for a remarkable number of people--even ones on this forum, where I often want to assume everyone should be an expert but in fact a lot of people just got into doing this work last week--that AI actually already has more experience than they do (and, now that they are using it so heavily to do their job, maybe ever will) and thereby makes fewer mistakes.

I don't know what that feels like, but it is certainly a very different experience than we are having. Maybe in another year or two, we'll have that experience also: if the AI gets good enough to become your peer, maybe we will find it interesting to use (or maybe we will become so demoralized, having not been in the position of having to work in a team of people better than us for so long--and these AIs wouldn't even be people!!--that we give up on the art entirely). I mean, imagine for a moment what it would be like if you found yourself in awe of the AI and were learning from it... I think it explains a lot of the confusing excitement about this technology.

I love it for writing tests. It’s good at picking up patterns, so if I need a bunch of unit tests then I can usually write one test and let it write the rest based on the RSpec descriptions.

I recently had a ticket that required refactoring some code that used net::http from Ruby’s stdlib over to using the HTTP gem. The rewrite was pretty easy to do by hand, but we had a bunch of test cases and rewriting them proved to be a bit painful because the mocks required were quite different.

So I spent half an hour rewriting the first test case with the new shape of mocks. Then I commented out the remaining tests and the AI recognized what I was doing and rewrote the remaining test cases with the new mock structure in seconds.

I treat AI like a junior programmer. I don’t trust it to do the hard stuff, but it’s great for scenarios where I have already established the pattern and I just need someone with basic coding proficiency to apply that pattern to a large chunk of code. It’s amazing at doing the grunt work.

Weird flex or whatever dude. But all I’m hearing is that your anxiety has led you to a) refuse to look into a new piece of technology, b) hold incorrect views on it (like that it’s only useful for boilerplate code), c) look down on those that ARE using it, and d) pose that your incredible experience is why this wouldn’t be useful to you.

If you can’t find your drive to look at new things enough to even see if they’re worth digging more into, which you seemingly haven’t, then I think this speaks poorly of you as a developer.

I'm sorry, but I do not get why you're making it personal and attacking my worth as a developer because I am doubtful about what Copilot can do for me.

It is possible to disagree without being rude and insulting.

I think you're speaking to a, most likely, young developer. Having dealt with many similar sentiments, a few more years usually smooths this out. I always wonder if tech work is good for certain personalities, it's quite easy to carry on with a misanthropic communication style and still succeed

Being polite to others costs nothing and makes this a better place for everyone.

Personal insults are particularly destructive to civilized discourse.

I tried it out, but didn't find it all that helpful. Maybe part of that is just getting used to it. I'm also worried about the long-term effects: I have a much better understanding of things if I write it myself. There's a kind of "practice" involved every time you write something (code, documentation, prose, music, anything) and I'm not so sure losing that is a good thing in the long run. Proof-read a thousand books probably won't make you a great author.

when designing, thinking, abstracting, we are actually not writing any code. I sometimes use a whiteboard for example, of course CoPilot cannot help. Once we have your designed, thought-through implementation in your head, while you are typing in an editor, CoPilot definitely can help you write code faster (or slower if the output is not exactly what you have in your mind). I see it is a productive positive product.

> CoPilot definitely can help you write code faster(or slower if the output is not exactly what you have in your mind). I see it is a productive positive product.

The actual act of writing code "faster" baffles me. Without tools like intellisense, etc.[0], I am perfectly able to write code as fast as I can type.

There's absolutely zero need for me to write faster.

[0]Intellisense, etc. tend to get in my way, so I turn it off.

Intellisense is a godsend when you are working in a large code base using proprietary API with varying level of documentation. It's very good at filling in the full name of that pesky class or giving you a quick list of functions in that package for you to pick the one you want.

I use it, for autocomplete. It’s great for creating the text for string constants, error messages, autocompleting if statements (which if variables are properly named, works great). Speeds up repetitive work but it’s not useful for actual problem solving.

I tested it, the suggestions were fine but not so helpful to be worth the property theft risk, so I uninstalled it.

You are not alone.

It is great for starters but my experience it everything AI generates needs to be double and triple tested and almost always manually modified to do exactly what you want to do. The 40% number just seems to be a tall claim and unless the data is shared and verifiable I don’t really give a crap about these claims.

I'm not worried. Automating our jobs away is part of what programmers do, and always has been, but somehow we never run out of work to do.

I have not yet grown curious enough about Copilot to try it out; it seems to solve a problem I don't really have, since my productivity is never limited by typing speed. As you say, thinking and designing is the hard part.

I can't afford to use it but if I did, I would use it as moderate as possible.

I look forward on the future of AI assisted coding tools, it could make us more productive on generating part of codes that we don't care much about, while leaving us time to deal with code that we do, manually.

There is a lot of annoyance in the boilerplate of a program - "set up a scaffold for this, template that, convert data from this format to that format". The AI tools I have experimented with are good at this and for sure this is useful.

But no one can read my mind! If I have to get increasingly specific in what I am telling the AI, and have to formulate my writing style in a way that keeps things very precise, I am essentially programming in another language. Albeit one where the output is uncertain and I have to check everything.

It is possible that someday it reads in an entire codebase (say, millions of lines of code), and I can describe a new API I need and it implements the entire thing. This would be quite useful.

> It is possible that someday it reads in an entire codebase (say, millions of lines of code), and I can describe a new API I need and it implements the entire thing. This would be quite useful.

You can get this today by switching to being a product manager/owner! Dream up an idea, describe it to a bunch of fuzzy intelligences, go through a few cycles of refinement, and voila: A software.

And now on to the other 99% of the problem. ;)

Only problem is that they cost $10,000/mo instead of $10/mo.

$/mo doesn't matter. $/function does.

That was my exact thought. I don't use copilot but I'd wager that at least 40% of the code I check-in is autogenerated as well. A fresh modern JS project is 15+ files full of boilerplate around a single "hello, world" statement.

If there was a metric for how much net new application business logic was written by copilot, that would be far more interesting to study.

I just started using Copilot and so far, it really does seem to save me time with rote stuff like imports, function definitions, and where I would otherwise be copying / pasting code I just wrote (say, while writing a bunch of tests). Even when it gets those slightly wrong, it still saves me time or at least makes it feel that way.

So overall I'm honestly pleasantly surprised with it. It feels similar to when I got a car with adaptive cruise control - suddenly I'm not thinking about maintaining the correct speed and following distance on the highway, which makes driving for a while require far less mental effort.

I know I probably could have accomplished something similar by setting up snippets or whatever, but I installed it and it immediately started doing what I wanted without having to ask.

Nifty.

To me, depending on the language, it's less useful than coc-nvim (too slow, breaks my tempo, too much added code, sometimes add stuff I did not ask for, or use old, inefficient patterns you only really find on shitty websites), so I still write the 'clever' part with vim, but when I write tests I now use VScode because copilot is really helpful (and it's funnier to use it than to write unit tests that work).

For languages I do not know well however I use it for everything. But I wonder if it keeps me from learning tbh.

There is a very good Copilot plugin for nvim written by the legendary tpope himself. It is easy to toggle on and off. I use it alongside coc-nvim every day.

https://github.com/github/copilot.vim

It would also be interesting to see how it does in less “polluted” programming environments than web frontend.

I am using Copilot and I sometimes have the feeling it does. About well 20 % of the time. 60 % is quite useful and about 20 % it gets in the way. But these mindreading moments are amazing

So far I’ve found copilot is like a weather forecast.

The shorter the prediction the higher probability of it being correct.

The longer, and more complex the prediction the less likely it’s all correct. But, with a close review you can spot the errors and correct it.

I’ve found most useful to get ideas on the page fast.

It’s also absolutely insane to be writing a parser, and have it suggest unit tests and code where it clearly has developed an understanding of the language before the parser is complete.

the generalization is errors compounding over time

You bring up an important point in general:

Copilot is great at writing tests.

I'm not surprised. A lot of the code I'll write in a day is trivial, calling helpers, writing test scaffolding, etc. I'm still building cool things the other 60% of time, but this job as we know it comes with a lot of repetition. (esp. if you write unit tests ;-))

unit test definitely a great use of copilot.

What are you talking about?

I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong.

Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it.

So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes).

Other people have said this, but copilot is auto complete.

You use it every time you press tab.

Do you accept an auto-complete suggestion 40% of the time? ...mmmm, yes, well, guess why 40% of code is generated by copilot.

It's not because people are generating tests with it.

Most of the time I know exactly what tests I want to write but it's annoying to write them. Copilot autocompletes, I check it and then I use it.

For me the worst part about this is that writing tests tediously makes me reach for layers of abstraction on my test code. And then suddenly my test code is complicated and needs its own test code, and changing a test can often be problematic to the abstractions I foolishly employed.

Being able to churn out the boilerplate for tests, which can make DRY a non-concern, is great. I just hope that if I do this, I never get sloppy, and I always review the tests.

I agree! I try not to make test code too complicated, and prefer repetitive code over abstractions (else you have to test the tests!) so Copilot is useful there.

In my case about 50% of the test code is just boilerplate, so I usually type the test name and generate the rest. Most of the time I have to rewrite the actual test logic (although very trivial tests are sometimes correct), but I probably keep more than 40% if we're talking tests specifically.

> In my case about 50% of the test code is just boilerplate

Why not replace it with a function call?

Some languages are just noisy.

Go, Java I’ve experienced are noisier than others.

A common occurrence is something like a pool of 10 actions where a bunch of tests each do 3 to 7 of them. This is very hard to abstract with a function call.

That makes the test harder to read. Tests should be dumb.

In the case I was referring to, mainly to keep the code consistent with what was already there and the PRs small, as the code base is primarily owned by a different team. It's also pretty innocuous short tests that read well as they are.

I use it for unit tests. That's one of its best use cases. Obviously I read the code it writes down.

In a recent personal C project (a custom archive file extractor), I have used chatgpt to generate tons of unit tests.

And honestly, I was extremely impressed. With a bit of context, it was able to generate almost correct test data for a fairly complex binary data format.

I've also leverage it heavily to:

* generate doxygen comments

* get usage examples of libs I never used

* create a whole bunch of utils functions.

And honestly for all these tedious tasks, it has done a far better job than I would ever had.

Comments were consistent in styling, the base examples were fairly good, and the utils functions were well made, specially the error handling (which I would probably have semi-consciously skipped tbh).

In fairness, I modified most of this code slightly to make it fit my project structure, or tweaked it a bit what the IA didn't quite get it right.

(it never fully understood some offset fields in the file format, but got pretty close to at times. And in fairness my naming for these offset fields was a bit questionable).

Heck, as a test, I even threw at it an RFC-like spec of the file format, and asked him to generate a python parser. The result was not 100% correct, but definitely a good start to iterate on.

In the end, this side project took me 2 weeks to implement with chatgpt probably saving around 1 week of dev. It also greatly helped improving the quality of the project (better doc, better tests).

> I don't know anyone who uses it for this

that would be surprising - it does a fantastic job of producing e.g. dumb unit tests. for instance, Copilot + a Go table testing template means you can churn out the simple "make a request to this url with this data, ensure I get a 200 and the response contains 'id' and not 'error'" extremely quickly. the code is trivial but tedious, so you can quickly inspect for sanity and run to ensure they pass, then commit and have them checking future changes.

> it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes).

variations on this comment are all over these threads, which is bizarre. hallucinating means you waste a few seconds reading the code it produced, not that you commit incorrect code.

this isn't like ChatGPT advising people to drink bleach, Copilot is a dumb tool offering an expert (you) suggested solutions for your expert consideration

I guess this could be true in some languages and settings, for example when a notable portion of your tests verify correct behavior with null values, arguments of wrong type, etc. When talking about unit tests that verify the actual logic, you should naturally become more careful with your copilot suggestions. Especially if copilot also wrote the code it's testing.

Yeah, to me this doesn’t feel that different than using a library from another developer, just slightly more customized. Most code that is run is not written by the final developer, but by people further up the chain.

The real question is, why we need to write "trivial" code? I think people have lost the plot there. We should be writing better compilers and libraries, instead of using AI that will write more code to deal with.

The obsession with code generation instead of creating proper abstractions is a scourge of today's programming. It's a regression in software engineering.

I agree to some extent. But there are redundancies that are inherently beneficial and make code more robust and readable, like separating interface from implementation (where some declarative elements have to be repeated), or explicit type annotations instead of full-program type inference. There are also strictly necessary redundancies, like when you pattern-match an ADT, you repeat its declared structure. For desirable or necessary redundancies like that, AI code completion can still save time.

Intelligent unit test creation (if it is indeed intelligent) is also a good use-case. Not to relieve the programmer from thinking, but to save typing and to point out potentially useful tests that you might not have thought of.

This is trivial stuff which is handled well by any competent IDE from 2015, and even a well enough-working set of editor templates.

I don't see where "AI" is required for stuff like this.

I agree, but also, this is actually where AI (whatever it is) would be helpful. Suggesting actual abstractions that could replace boilerplate. Of course, these can build on existing abstractions.

The boilerplate makes it readable by humans.

I have made the same point elsewhere to AI enthusiasts friends of mine.

The fact that defining the same task in natural language to an AI is easier than writing it in formal language, means the formal language lacks expressiveness [0], vocabulary, or more likely, both.

But if they are both at same level of expressiveness and vocabulary, formal language wins, because it can spot and prevent ambiguity.

The question is how scalable it is to implement such vocabulary into formal language.

[0] - The line is often blury. How many languages allow to straightforwardly express `clamp m between 1.0 and 2.0`? You can argue this is vocabulary, or this is expressiveness.

> How many languages allow to straightforwardly express `clamp m between 1.0 and 2.0`? You can argue this is vocabulary, or this is expressiveness.

And I think it would be a great place for AI in programming - automated refactoring. For example, it would see a code that clamps the value and could be replaced with a library function call clamp(m,1,2). (And it would also, presumably, made sure that the code is indeed the same.) So it could help you understand the vocabulary.

Or, if it sees you are doing this frequently, and it's not in the standard library, it would create the function.

But instead, this AI (more like artificial stupidity) will just suggest you the same thing you wrote because you don't know any better in your code base. That's not helping.

Good point. But I think current approach of large scale generative AI models is not sufficient for this.

Sometimes it is just the syntax of your language. We can't all be writing lisp where the syntax is just another abstraction.

> We can't all be writing lisp

Why not? Isn't it easier to learn new syntax than to figure out a correct prompting for AI? (Although actually, Haskell syntax would be something I would prefer to write in.)

> We should be writing better compilers and libraries

this is a bizarre claim to make and just completely ignores reality.

people ARE trying to make better compilers and libraries, and have been for decades. we are we are now after person-millennia of effort, and that still means the optima we're at requires writing some repetitive web template rendering code in JSX or whatever, which Copilot can automate. why didn't some homomorphic-monoid-transformer-auto-renderer win instead?

I think the reality is that "if you build it they will come" is simply not always true, and worse is sometimes better. I don't think we are climbing to an optimum, at least unless it is culturally perceived that good abstractions are more valuable than just generating code.

The auto-renderers did actually win. There were GUI building tools at the end of 90s (like Visual Basic) that still put most web frameworks to shame. But, for various business reasons (mostly pricing, really) they were replaced with "web technology" - something open source but worse.

(And I am not dissing OSS, either. For example, Qt/KDE had some great abstractions that could serve as a basis for building applications. But the urge to reinvent the wheel has won.)

I guess one difference is that Copilot can guess things that it’s not sure about and have you check over it, but a compiler can’t just guess what you want it it would be very unreliable

Compilers do however guess that you don't want something - it's called compiler warnings. Linters do that as well. To me what you describe is more about how we communicate with compilers than an inherent limitation.

I couldn't agree more with everything you wrote. The problem with repetition is not that it's hard to generate, the problem is it's hard to maintain!

I've used ChatGPT to find libraries and to help me line up types. It was very helpful.

Sure this must be using the most generous possible interpretation of the data. In my IDE the suggestion is shown as an autocomplete option on basically every keypress. I will only hit tab to insert if it is what I was planning to type. Sometimes if I really want to use it I'll coax it, write a comment or edit my code a bit to give it some help until the suggestion is what I want. There's no way they're counting every autocomplete suggestion in the denominator of that percentage.

It's 40% of submitted code, not 40% of suggestions.

I guess you're right, I initially read it as "40% of AI suggestions don't get modified", not "40% of code is AI generated".

I had it on yesterday and had the most miserable afternoon's coding I can remember for a while.

It was a particularly tedious, ugly, and cludgy bit of work that needed doing. Pulling in a data from several very broken sources with horrible naming conventions, patching holes and generally trying to make a sane dataset out of nonsense.

I'm convinced Copilot made the whole process ten times harder than it needed to be. It was constantly pre-empting me with plausible-sounding variable names and merges which led me off down the wrong path a few times.

I flick between having it on and off, and recently I've found I'm much happier with it off. Great for docstrings, boilerplate, and stuff like "recursively traverse this json for [key] and return [value]", but for day to day stuff it's a pain in the arse.

A lot of the code it generates is trivial, or 1-2 lines of code that's obvious, at least for me. In which case, it makes no sense to modify it. I have a feeling this is the bulk of it.

This is my experience as well. For longer code, Copilot introduces errors through making up variables and method names that don't actually exist on the object. These errors are very annoying but only 5% of the generated code.

I tried Copilot last month and canceled my subscription because of all the small errors in generated functions. It was more painful to debug than write it myself.

Maybe I should have lowered expectations and thought of it as improved code complete. I got spoiled by copy/pasting from ChatGPT. As it stands now I'd rather pay for a subscription to have the latest ChatGPT than for Copilot.

I feel like it has been getting worse for me in the past few months. I will write a comment and it will autocomplete - but have something wrong - almost like using chatGPT to try to generate code. Mostly right - but not quite. Things it would have gotten in the past.

Maybe it's just me.

Read: 40% of the code Github Copilot users check-in is either boilerplate or has bugs in it.

true that. 40% of the code is AI generated doesn't mean any of it actually runs. Morely likely we're on the hype cycle of AI generated code right now.

How is this different from Google searching then copy and pasting from Stackoverflow? Software is hard because of the 20, not the 80.

A. Direct solution vs choosing from 4-5 stack overflow answers. Gives an illusion of doing your work for you.

B. AI can slightly manipulate existing code according to your prose. But I have also seen it fail hilariously to follow basic instructions. Eg: without `All argument constructor` in - but it still generates the all argument constructor. Another day it writes regex but fails to escape some characters properly.

All of you should stop posting your knowledge of programming online (Hacker News, Stack Overflow, Reddit, blogs, etc) unless you want an AI to absorb it and take your job from you.

I guess they should primarily stop using GitHub as a source repository.

So let’s have a test where two or more teams build the same software. One set has AI tools, the other doesn’t. The software definition is provided to both at the same time, and there is a set of tests to see which team produced the software the fastest and produced the highest quality software.

I'd watch that game show.

Call it The 80-20 Show. Compared to the not-AI'd team, the AI'd team gets 80% of the correctness in 20% of the time. Then it is revealed to be an application where errors are life-threatening. Thrills! Dread! FUN!

The boilerplate code problem was solved by Lisp in the 1970s but we keep insisting on coming up with worse solutions because ‘parens look funny’

Mostly because macros are unintuitive. But look, higher-order functions went mainstream a few years ago; immutable data structures, too. Sum types claw their way into mainstream, first to replace null, then for error handling. The industry is not hopeless.

(are (excessively funny) (serialized (of (parse tree) structures)) though) ;; this limits direct Lisp adoption. Python and JS / TS took its place now.

IMHO, if you really "tell a CEO I can give him 56% more developers overnight" , he will sue you. Every programmer tried it out can tell that his affirmation could be right only for trivial part of a program, for non trivial programming part, it is more like a board game, where you try to induce the "AI" to produce the right answer modifying the "prompt" again, again, again, again, ( Did I tell you "again" ?) and again. <sarcasm> Obviously that is a customer fault because he didn't hire prompt engineers to do properly the job to describe properly the job they need be done </sarcasm>

I mean this is a prime example why lines of code are a terrible metric.

I think it’s more interesting that they track this.

I’d be more interested to see the other stats..

How much code is AI generated and then modified.

How much modification is typically needed.

How often suggestions are not used/abandoned/deleted.

Copilots positioned to optimise what i call "Google developer" flow. These are commadonity devs every workplace has who dont really know how to code, but rather know how to ask google questions to get the code they can slap together to meet a AC.

On the other side you have devs who have AI in their head. They often use tools to optimise their flow. Ive always used resharper as a developer and think autocomplete is certainly a developer efficiency improver. 99% of the suggestions resharper makes, i checkin. Thats why i use it, it has less fuzz because it doesnt try to be everything.

Pressing alt+tab and scrolling through some search results isnt something that adds overhead to my flow - after 30yrs im pretty good filtering content.

I do alot of PoCs though, so if i wanted to write in a language i didnt understand, i think copilot can improve my flow somewhat to get me to serach terms that will benifit me quicker.

Overall im still a few years away from relying on AI to help me code. Its going to be great for retirement though, if i had a family and kids it would allow me to checkout of self-paced learning while still maintaining my edge.

So how long before exploiters train code completion ais to generate back doors in the mountains of other boilerplate?

I am a PM and a novice programmer. Unless copilot improves drastically for me, I am going to cancel the subscription and keep the one for chatgpt pro for now.

I have been able to write fairly decent programs using chatgpt, am starting learn and build full stack apps etc. I am not able to use copilot. Maybe I do not know the tricks.

You have to get Copilot started, writing a comment is often enough to do so. It doesn’t write from prompts like ChatGPT of course.

I just added an Ask AI feature to our product to help customer who want to write inline javascript functions to modify workflows with code instead of UI... so far it generates amazingly good code with a realtively simple system prompt.. that just let's the AI know ahead of time details about our API such that when a customer says: "Write a function to search for a lead in salesforce with jsforce and update an activity record with the duration of the call." It can really generate code that does just that... I do have to tell it to use a specific jsforce library or it'll just generate non working code... but IMO that's unreal and amazing...

That question generated this code: [I did not write anything beyond this point]

Here's an example function that searches for a lead in Salesforce using jsforce and updates an activity record with the duration of the call:

``` exports.handler = async (event, context) => { const jsforce = require('jsforce');

  // Create a connection to Salesforce using the access token and instance URL
  const conn = new jsforce.Connection({
    instanceUrl: event.salesforce.instanceUrl,
    accessToken: event.salesforce.token
  });

  // Search for a lead with the given caller number
  const lead = await conn.sobject('Lead').findOne({ Phone: event.activity.caller_number });

  if (lead) {
    // If a lead is found, update the activity record with the duration of the call
    const duration = 10; // Replace with actual duration of the call
    await context.ctm.update({
      activity: {
        id: event.activity.id,
        duration: duration
      }
    });
  } else {
    console.log('Lead not found');
  }

}; ```

Note that this function assumes that the caller number is stored in the Phone field of the Lead object in Salesforce. You may need to modify the code to match your specific use case. Also, the duration of the call is hardcoded to 10 seconds in this example, but you should replace it with the actual duration of the call.

lol I use co-pilot and there is no way this stat is true.

In my experience it's more like 80-90%, given the right prompts. Very little needs to be changed, usually

I'd say closer to 40%. You almost ways need to adjust the parameters of function. Almost always need to rename variables, etc. And this is writing mostly boilerplate too.

For more custom stuff, I mostly just ignore the suggestions since they are mostly useless.

> 40% of the code GitHub Copilot users check-in is AI generated and unmodified

> there is no way that stat is true

> I'd say closer to 40%

What?

Technically 0% is closer to 40% than it is to 90% although it would be weird phrasing.

I wonder if it’s 40% of the modules,40% of the functions, 40% of the lines or 40% of the tokens.

The first would be very impressive, the last not so much.

Not sure if I’m using it wrong or the problems I’m coding for aren’t totally generic. The best help I get from it is a little string formatting or some basic error handling etc. I actually cancelled my subscription earlier today because I’m not getting enough benefit from it.

I'm thinking about this in the context of GPT-4 going from bottom 10 percentile to to 90 on the LSAT and similar standardized tests. What's going to happen to this figure when they get bigger and better models to drive Copilot?

This is double risk:

1. Security vulnerabilities from unaudited code.

2. Copyright violation risk, since a lot of CoPilot examples show clear excerpts from open source repositories.

AI is definitely going to revolutionize the industry! I'm excited about replit, marsx.dev, and github copilot.

for me it’s a fancy autocomplete at its worst! it’s a handy tool to cut down on repetitive tasks

This is exactly how I use it also. It's great when it suggests the correct thing, but doesn't interfere with me if it's wrong.

99% of that code is stolen.

Well done Micro$hit.

Reply on news.ycombinator.com