Italy did an accidental experiment on the effectiveness of GitHub Copilot | Hacker News Reader

Italy did an accidental experiment on the effectiveness of GitHub Copilot

arxiv.org

23 pointsdadadad10016 comments

I wonder if the 50% drop was because developers were half as effective without copilot. Or was it because they have spent half of the day looking for ways to bypass the block while having angry conversations with colleagues, and therefore, were not coding.

I was interested to see how it affected them in the medium-term, including the VPN/Tor usage, but the data range seems extremely short, a few days before the ban and a few after. Also doesn’t seem to account for any other events taking place in and around that time, or that it’s a weekend and so on.

Doesn’t really say much, imo.

The idea that aggregate developer output is 50% improved by the simple existence of transformers seems to not pass my sniff test.

I haven't seen any of my coworkers improve their output in the past x years.

It's easily true for me. Far more than 50% for someone doing simple React development or DS scripting.

I still think that if your productivity is seriously impacted by chatGPT, you are easily replaceable as a programmer.

I’m curious what job you are doing that you don’t feel chatGPT can help with?

Talking about Copilot, not ChatGPT, but it saves so much time when it comes to boilerplate code, naming variables, writing comments/documentation/descriptive error messages. Things that are generally easier to read and verify than to write. The ability to do these things is not where my value lies as a programmer.

I mean that's kind of silly.

Everyone doing "hard" work, whether it's a software engineer developing novel algorithms or a PI at a lab researching open questions spends at least 80% of their time grinding through operational necessities. Writing tests, writing grants, debugging... nobody is sitting a a chalkboard 100% focused on the bleeding edge.

GPT turns that 80% into 40%. That's a lot!

The link says nothing about Copilot, this title is not just editorialised but wrong.

@dang: this post is marked as flagged at the moment, but I don't think it should be? Except maybe for the editorialized title, but that doesn't seem super bad.

OP here. Happy to change the title to “on the effectiveness of ChatGPT for GitHub users”. Sloppy reading on my part. Sorry

> Release event

I don't understand this. Did they measure the number of releases in the GitHub "releases" feature?

By turning off access to GPTs the Italian government showed how their country’s developers felt about copilot

Like with most event studies, they don't control any other events that might possibly affect the likelihoods. It seems that the ban also occurred in a weekend. They should also show the raw time series from a longer period in order to see how stable the series are and thus how realistic the effects are.

Copilot was not included in the ban, only ChatGPT. Flagging.

Yep. Sloppy on my part. Suggested better title above but I can’t change it myself.

Reply on news.ycombinator.com