Coping with Copilot
sigarch.org
sigarch.org
One thing tho is that things like Copilot put the lie to the hypothesis (propagated mostly by the Google-style job interview) that intensely coding clever for-loops to perform algorithmic magic (for things usually already in the standard library) is the best measure of the competence of a software developer. If it can literally be done by a machine, maybe we should be measuring based on something else. Especially since this kind of thing is best done on the job by looking in a good textbook or using the standard library. Probably grading at the university level needs to consider this also.
I haven't used Copilot. I doubt I'm its intended audience. After 30 years, actually writing code is perhaps the easiest part of my job. The mechanics is the easy part. The big picture thinking and figuring how to get it all together into a system is the hard part (and honestly there are plenty of people far better at it than I am). Now, if we get a Copilot for that ... then our profession is in trouble.
Here's an example I just now tried in the codebase I had open. I literally typed in the following Go in a function:
mean := geom.Vec3D{}
for
mean := geom.Vec3D{}
for _, detection := range filtered {
mean.X += detection.X
mean.Y += detection.Y
mean.Z += detection.Z
}
mean.X /= float64(len(filtered))
mean.Y /= float64(len(filtered))
mean.Z /= float64(len(filtered))
I'd rather see Copilot used to expand existing libraries. Pulling potential additions to your own library off of Copilot would be an interesting twist on the situation. A hacktoberfest-alike based off this would be weird.
I think you should give it a try and see what you think about it. I was hesitant about it at first but was very surprised at how much time it could save me from having to look up things on Google. I've found it especially useful when I'm switching to a language I may be less familiar with. I can understand the basic logic of what I want to do, but would have to spend time looking up how to do it in this specific language. Orrr... I just have copilot help me out and generate such a solution.
It's obviously not going to be a tool that is applicable to everyone. Just like how many manual labour oriented contractors have a bunch of tools, each of them may have their own set of tools that slightly differs from the other person. That is okay, and there would be no need to try and bring someone else down for their choice to use a certain tool.
My concern is that I'd get locked into a specific IDE.
Besides, the generated example seems to be missing code to gracefully handle the case where len(filtered) is zero. Maybe there's a precondition that prevents that from happening or maybe a division by zero is exactly what you'd want, but at face value it looks like the bot did a rush job.
Saturation is not. This is what really bugs me: If I'm going to drag in a billion GPUs of external computation (or a dependency, which is basically the same thing but with human brains), I want it to provide the hard algorithm I can't write, not the easy one I can. I am not limited by typing speed.
I also wonder what a "detection" is.
(Also, in--say--Ruby and JavaScript 1.0/0.0 is Infinity and not NaN.)
Both good examples of coding where you should be thinking instead, though.
For what it's worth, Copilot (correctly) inferred a loop variable called "detection", I imagine based on similar usage earlier in the function. And there is already a conditional in place to prevent invalid operations; if I remove it I see a new suggestion:
if len(filtered) > 0 {
It automates a great deal of boilerplate crap that you have to do especially in web frameworks such as angular.
Also I work in Rust these days, and use CLion. Not sure if it would be much help in that environment.
Since Microsoft uses material for copilot outside of licensing on the basis that it is Fair Use, that would probably have no effect in practice on whether or not the material is used in training something like Copilot. For that to matter, you’d first have to win a lawsuit on the basis that training something like Copilot requires permission of the copyright owner of the training material, to invalidate the premise of Microsoft’s action.
i think that any formal violation of an open source license would be very bad in terms of public relations - even if they can claim fair use.
I think that's the part that people who don't think it's worth the money aren't getting. This kind of system is godsend for the likes of Infosys, TCS etc. So the immediate threat is to the jobs there - but the side effect is that it'll make it all even cheaper, so we'll see more "outsourcing to the cloud", so to speak. Often to the obvious detriment of quality, but that doesn't seem to matter in this market.
Anything generated by copilot, which if it is a derivative work, is not something that copilot can hold the copyright on.
From the auditor's perspective it doesn't matter if you copied it out of stack overflow, from some GitHub search, or copilot. You, the human, didn't check the license / plagiarism detecter. It is you, the human, claiming copyright on the work you are creating which may incorporate material from other sources.
Copilot isn't claiming fair use.
You could argue that the model that copilot runs from is a derivative work (and this is going to be interesting when it gets to the courts, because, frankly no one will come out the 'winner' on this when trying to explain it to a judge) - but that's not the code that a human is claiming to be their creative work and is ultimately the license violation.
Personally, I (not a lawyer), believe that copilot is on ok ground - but anyone using it needs to do their due diligence in verifying that the code that they've incorporated is licensed appropriately - just as if they've copied something from Stack Overflow - who knows where that copied was copied from.
I have less concerns with identifiable code from copilot than humans not caring about the licenses of their source material in creating human generated content.
The moment a fragment of code from a GPL'd or AGPL'd project shows up almost verbatim in someone's closed source or non-copylefted, etc. project, and someone proves it, sparks are going to fly.
And it's probably already happening, just people haven't discovered it yet.
How many years did the Oracle/Google lawsuit go on for? And in the end about a handful of a lines of code only tangentially related to the issue at hand?
Part of the lesson from that must be: employers should be telling their workers to say TF away from Copilot or things like it. And be careful in general when browsing source. License literacy is critical.
I don't touch it because I need to feed my kids. I don't need my career exploded. Overcautious? Maybe. I'll let someone else find out. I make a living in and around open source software.
I'm hoarding popcorn and can't wait for that, honestly.
> License literacy is critical.
It's beyond critical, but most people I have talked says that they see the right to copy and use any code they see online. They don't care.
Building this open source corpus was not easy, and we need to defend it too. This is a culture.
ISTR that Amazon's version of copilot has licensing info output too.
This fact solves many "we need this lib but it's too open" problems. The author can make a special license just for you if they want.
It is the nature of the beast, however I make my stance clear and stand by my principles. It is equally important for me.
It's not even clear how often training machine learning algorithms on code results in copyright violations. CoPilot does have a setting to detect and disallow direct copying, but how well does it work?
This legal uncertainty is enough that I wouldn't advise using it, but maybe people who use it will be fine?
Europeans will probably preempt that with legislation.
The phrase “derived work” is, IIUC, a phrase from copyright law. And you’d have a hard time convincing me that Copilot-generated code is not a derived work from its training data.
> And given that they’re open source code bases I expect licenses to explicitly disallow things, and consider anything not explicitly disallowed as permitted.
That is very much not how copyright and licences work. Copyright law gives the copyright holder the exclusive right to make copies of the work, making derived works, (and to do some other related things, like making a public performance of it, etc.), so to do any of those things, you need explicit permission, i.e. a license from the copyright holder to do it. A license is not a list of things you are forbidden to do; on the contrary, it is a list of things you are permitted to do, which you would not otherwise be legally allowed to do according to copyright law.
This is a novel scenario. It seems unclear how the courts will interpret it? Never mind what we think, will they decide it's a derivative work, or is it a transformative use?
Suppose we create a new AI image generator, and use as training input every image ever made of a Disney character (official images by Disney, that is, no fan art), including every frame of every Disney movie. Could we just use the output images of that AI however we wanted to? (Not withstanding trademarks.)
https://en.m.wikipedia.org/wiki/Copyright_protection_for_fic...
I just don't understand the OSS community sometimes. "Software should be open and free (libre) for me to study and modify" includes what Github did for copilot. If you don't want your software to be free (in either sense), don't host it on an open source platform, especially one that makes it available gratis to the public.
There's possibly a valid argument that any private repo code that was used for copilot doesn't fit the proper definition of "open" (or gratis). But I haven't actually read the Github license around this, so I don't know.
Copyleft, free software, GPL style licenses do not have their source open purely for the purpose of studying and modifying. Their licenses also require that derivative works also be free and that such modifications be distributed.
Copilot does not comply with this. And so violates the spirit of those licenses, and probably also the letter of the law.
And it’s at best questionable whether copilot is derivative of the code it is trained on.
And by that logic, gpt3 and dall-e are also copyright violations. Except this has never been proven to be the case.
The completions are often more “similar to stuff I’ve typed before” rather than “generate this working function”, and it’s not an order of magnitude improvement over just regular intellij completion.
…but I find it’s generally better.
Up to you if you consider it worth the cost of paying for it, and the macro completion is so-so, but “its rust so copilot can’t do it” isn’t really true.
Until then, copilot is a giant liability that ensures I can't use it for code that my company ends up owning, nor can I contribute code I write with it to literally any open source project because in a very real sense: I didn't write it. I just assembled it from parts unknown, and those parts may end up being lawsuits.
Obviously IANAL so this is largely conjecture, but until we _actually_ see how this would play out in court, I'm leaning towards this being less of a legal issue than folks here act like. For personal projects, I'd say the likelihood of some other engineer reading your code, noticing a similarity or duplication, and dragging you to court for it is near 0.
- "you were hired to write code for us, not to use an autocomplete service that makes us liable for both copyright and patent lawsuits, I hope you like getting fired."
- "as per this project's license, we can only take code on board that you contributed under our license, but an audit shows that your PR/MRs contain tons of GPL/MIT/Whatever licensed code instead. We're going to have to back all of that out, and we're going to revoke your contributor status"
- etc.
If you don't know where the code in your autocomplete comes from (and copilot can autocomplete large swathes of code) then literally anything that comes out of "you didn't write this code" may apply. From fraud (depending on what contract you signed) to trademark infringement, to license violations, to even just simply misrepresenting your skills to an employer. As with all things, it's a sliding scale, but just because the majority of incidents will be on the bening part of the spectrum doesn't mean the litigating part doesn't exist, and that's what your legal department plans for.
Work for a big company? Good bet you're not allowed to use copilot. And depending on the company, not even "for your personal projects" because you might accidentally read someone else's license encumbered code that you would not have come up with yourself and may now open your employer up to "you stole our ideas instead of properly crediting/paying for licenses".
Copilot is a legal nightmare.
I only used it for a language I'm very familiar with. I'd be a lot more hesitant using it for a language I'm less familiar with because I won't be able to spot the problems so easily.
I wrote the grammar in a comment at the top of the file, wrote and imported the AST enum, and wrote the first production. After that, I just prompted Copilot and it worked its way down the grammar, producing the parser functions one at a time. The CLion integration was able to consider the imported data structures as part of the prompt, so it even stored everything in the correct AST nodes.
For something like that, it's easy to verify that it did it correctly (through visual inspection and testing), and it allowed me to write the entire parser in about 2-3 seconds per rule.
That's a very limited, and dismissive view. If you allow students to use copilot, you're handing out certificates or diplomas to people who can't code. That will very quickly erode the value of your institute's certification, and with it, that of the other students. Otherwise, why not give everyone an MSc in CS? "It'll only hurt themselves in the long run".
> coding clever for-loops to perform algorithmic magic
A for loop for searching or summation is not algorithmic magic.
> maybe we should be measuring based on something else
No, why? Students have to understand the fundamentals. What those fundamentals are depends on the expertise. Once you've shown mastery, you can use more advanced tools. Teaching has always been like this, and for a good reason.
Copilot mostly helps write boiler plate. It can also do a good job with adding correct comments to that boiler plate.
It's a tool that saves time and repetition. It's not a tool that's writing algorithms or solving problems for you.
If everyone was great at coding after a year or two, you could do a lot more algorithm analysis,theory, engineering principles, mathematics and other sciences, etc.
Very true. Like it or not, the reputation of educational institutions plays a significant role in candidate selection processes at many companies. If a company brings in 10 graduates from a local college and discovers that 7/10 of them can’t actually write basic code, they’re going to become more hesitant to bring in other graduates from that college. Meanwhile, someone who has Stanford or Harvard on their resume is virtually guaranteed an interview. Reputation matters far more than I ever thought it would.
The funny thing is though - the CS programs at Stanford or Harvard are more research based than anything and hardly teach students to code. You are forced to take maybe 1 or 2 "learn to code" courses and everything else is just math/theory or assumes you can code anyways. And that shouldn't be a surprise; it doesn't take 4 years to learn to code; there are plenty of 15 year olds who can pass leetcode style interviews. Everything else you learn is far more valuable (although it's not a given that you will get a chance to apply it).
Are you sure?
The diploma is not issued after only the 101 course.
The problem an earlier comment expressed is that the student will sabotage themselves.
What else did you think that means besides exactly what you just said?
They sail through the beginning and splat in the middle, because they didn't have to learn anything in the beginning.
My comment is reponding to one that says the problem is that students get diplomas witbout learning.
I am saying that diplomas and certs are not a problem, not that there is no problem. The problem is what the earlier comment said in the first place.
Actually I'm not even sure that's a problem either in the long term, but it is the immediate problem. It may just be during a transition period while schools have not yet adapted to reality and are still worried about calculators in math classes. But at this moment, it's a problem.
Certainly there are a lot of pros and cons to getting a BA in CS. But "learning how to write code" is not the major advantage of a university education, nor should it be.
I think we can all agree that whether a company thinks something is good is completely unrelated to whether that thing actually improves your programming ability.
From downvoters: I would genuinely love to hear a defense of an MSc per se. What university has their shit together enough to actually accelerate/compress 2-4y of work experience into 1-2y years of academic project, which is what's always the claim? If all you're doing is the courses, those are accessible to 4th year undergraduates in every university I know.
To a person, everyone I've interviewed who did an MSc for the coursework had to because they finished their BSc with minimum effort and no one was interested in hiring them. Even worse is when you see 4y BSc, 2y work, 1y MSc.
There are a lot of people coming with 4+1y or 4+2y purely CS/SE. That's who I'm talking about.
If you're just looking for a code monkey to do your e-commerce react website then obviously work experience will always trump academia.
For example, I used to work in our IT department that handled the CS/Math departments at our University. We are a fairly well regarded school in the computing fields. However when we did co-op hiring of students for roles we had in the department there was one thing that stood out. People who only had programming experience through the school were not good programmers. It was the people who either had past work experience or had their own side projects who actually knew their stuff well.
At the end of the day I think CS degrees often go heavy on the theoretical and I find much of the theory is not applicable to the majority of CS/SoftEng jobs out there. I feel like someone who is motivated can gain the skills to be hirable in many software companies out there without going to school. I view things like co-pilot not as a problem. This will hopefully force the professors to make assignments that teach people more complex and work applicable content, rather than having them write generic intro to programming style cookie cutter programs.
I don't see an MSc accelerating anything; in general one year of MSc seems worse than one year of work experience or one year of motivated self-study, and the students who do it were not the top tier of BSc students.
Yeah, no. This kind of nonsense filling it up is precisely why an MSc in SE is useless.
Seriously though, you clearly need a reality check. SWEBOK is by no means necessary; it’s far from it. Saying it’s a complete waste of time shows that you have poor judgement on the subject. To your earlier point, having someone guide you through the poignant aspects of CMMI 1.3 is exactly the kind of value added by paying a teacher (where you would otherwise have to sift through piles of less valuable material). I only preach out of a sense of trying to help you and the others you have influence over. If you’ve actually worked yourself into a position where you hire people, and you’re not just pretending on the internet, you could do yourself a favor by being a little more open minded. Maybe consider that your original reply has been downvoted on a forum with “hacker” in its name.
Maybe that's why you're getting downvoted. (Not by me, for the record.)
But for MScs, nah. The more focused coursework is directly accessible to undergraduates, and the projects are often like "port this 20yo tool to a new kernel version" or "build me a DB schema" or a few other things you'll have also done but 3x over in your first year of a real job.
This seems to suggest that asking a candidate with an MSc about their thesis might be useful
Many master’s holders started out thinking they might have been interested in research and decided industry was a better fit after seeing how the sausage is made. You’re “rule of thumb” punishes undergrads who were inspired to push the boundaries of knowledge in the field.
Your explanation can be reframed as "many MSc holders did not invest any attention to how the academy works during their BSc years" - why should I prefer them? My rule of thumb punishes such people compared to those with a broader view and better self-knowledge, yes.
More inferiority complex. Some people give a shit about research and want to give it a try to see if they can make it work. There is no way you understand the nuances of how NSF or DoD funding works for your specific areas of research until you spend a lot of time in the weeds of grant proposals and reading reviews.
> My rule of thumb punishes such people compared to those with a broader view
This is a joke. Most BScs have no clue at all how academia works because they don’t care. They are absolutely not more enlightened to the nuances of career prospects in research in their areas of interest.
There are wide gulfs in the funding available depending on sub fields of CS. Research security and you’ll have plenty of money and opportunities at national labs, research arms of corporations, and academia. Go the P/L route and it gets dry pretty quickly.
> Cool personal insult - who has the inferiority complex here?
If you’re gonna try to insult, at least get it right. It would be a superiority complex if you wanted to view my post through the lens of pushing “how smart” grad students are. Yours is an inferiority complex because you’re taking out your lack of institutional credentials on anyone who has them with some convoluted logic about how people are stupid if they spend extra time in academia.
At any of the large CS universities, there are usually 4+ masters tracks you can go down (security, hpc, p/l, HCI, algorithms, etc).
Someone who quits with just a BS has no interest in going further theoretically. The absolutely did not cover all of the topics available - let alone get into research.
The most legitimate use of a MS degree is to move from one field to an adjacent field (eg math to CS). You know the basics, and you just need the advanced coursework in the other field.
A lot of people I have seen use a master's as a pure signaling tool, getting a high-profile University on their resume. For example, I have seen a lot of resumes with "BS from Unknown University" and "MS from Stanford/Harvard/MIT." In interviews, these students have not performed nearly as well as students with a BS from Stanford/Harvard/MIT, so I tend to discount the MS if I see this pattern.
Another common reason for the MS that I have seen is as an immigration tool. These people are usually pretty smart.
I never thought it would be held against me though. Seems weird.
I think the uncharitable view people take of a Bachelors followed immediately by a masters but with no PhD is that they presumably weren't planning to go for a PhD, and given that, had the choice between a) doing 2 more years of school for very tenuous benefit and b) going directly into a job where entry level pays 1.5x+ the median household income in many states. And they chose a).
So I think the uncharitable interpretation is then that the person was afraid of entering the adult world or felt like they couldn't hack it.
However, I see this as a failure of CS education. Our work is with computers. If we do not teach, how to use the tool we work with properly, then we have failed. I know CS is not only developing software, but also a lot about the theory, but it should not exclude practice of working with the most commonly used tool, which graduates will use, the computer.
Often even things are taught wrongly (do not get me started on "every noun is a class" kind of teaching of OOP), by people, who have not developed a single larger code base for decades.
That said, the industry has at least in that regard caught up a bit, and realized, that fresh from university most people do not write good code immediately.
CS fundamentally is not about computers. It's about math and the flow of information between theoretical machines. Computers just happen to be a physical implement.
The idea that universities should teach to develop industry-fit individuals is just an extension of commodifying university education. Why not let the industry handle training for what they need their people to do, and let universities focus on their intended goal: create more university professors/researchers.
What about moving the responsibility (and financial risk) off students' backs and to the corporations that want to exploit the knowledge gained in a degree through apprenticeship and learning to code through experience?
Which would result in universities eventually going out of business. An outcome I'm not opposed to, provided what replaces them functions better at training young people for real life.
I doubt, that a computer with its endless possibilities of using it and programming it, is comparable to a telescope. Also I did make note, that CS is not only about computer programming. So I am fully aware of the following:
> CS fundamentally is not about computers. It's about math and the flow of information between theoretical machines. Computers just happen to be a physical implement.
However, this does not mean, that universities should forego teaching about computer programming properly.
The alternative is, that we acknowledge, that first we study CS, but then employers finance year long computer programming classes for new employees, taught by people, who are actually good at this. The kind of people, who have lots of experience with different systems, programming languages, paradigms and teaching in general. That'll be expensive.
> What about moving the responsibility (and financial risk) off students' backs and to the corporations that want to exploit the knowledge gained in a degree through apprenticeship and learning to code through experience?
Yes indeed, that is one alternative.
I am saying, that employers should not expect and of do not expect good good to come out of people, who recently graduated from university and have their BSc or MSc. I don't think we are in disagreement about that point.
However, realistically this might lead to some employers being to much up in the clouds, only ever accepting employees, who were taught elsewhere and students having a really hard time finding _any_ job at all, because no one is willing to expend the resources to teach them. Furthermore, the teaching will probably be very one-sided, gaered towards what that particular employers uses in their tech stack. It is also highly unlikely, that most employers will have teachers, who are really that good, lets say up to Abelson and Sussman standards, if we may wish for something.
CS departments are getting more students and funding in recent years precisely _because_ they are seen to be providing valuable education for would-be programmers and practitioners in the software industry. At the same time, some people like you perpetuate the myth that only the theory matters in a CS degree, and it shouldn't have any bearing at all with the demands of industry.
If what you propose is becomes reality, expect CS departments shrink to 10% of their current size and high school graduates skip university to take on apprenticeships. That's actually fine by me, but I doubt that's actually what you're envisioning.
The fact is that most universities these days _pretend_ that they can prepare somebody to become a software engineer by providing CS courses. That works to some extent, but as you say, the degree is generally more geared towards training researchers and academics. There's no fundamental reason why universities shouldn't provide a more practical-oriented "software engineering" degree of sorts, much like how they provide other engineering degrees (eg. Civil Engineering) as opposed to pure sciences. _Except that they generally aren't capable of doing so_, so they happily accept the sub-optimal arrangement that CS is the de-facto degree for prospective software engineers, while maintaining the whatever perceived high ground where they insist on teaching theoretical, impractical stuff because most of the faculty don't know a thing about developing software in modern environments. Basically they want to have their cake and eat it.
Software engineering has become such an important part of modern society that IMHO it should at least have the recognition and respect as such. Imagine somebody arguing that universities shouldn't provide Civil Engineering courses because, you know, it's just physics and applied maths, and civil engineers should just do a Bachelor of Science in Physics with a theoretical focus, leaving the practical knowledge of how to build a building to apprenticeships, because universities are too busy mentoring the next Einstein. Would you want to live in a society like that?
A Masters in CS doesn't tell me the person knows how to code well, but it does tell me that they are _more likely_ to
- Have had experience with a wider variety of algorithms, data structures, and general coding concepts
- Have written code to perform more complex things
- Written code for someone "outside of a classroom"
Certainly the last two are more directly accomplished by actually being in the workforce. However, an educational environment is a far better source for the first one. Now, I'm not saying having a Masters is necessarily _better_ for a developer but
- Given two developers with no work experience, one with a Masters and one with a Bachelors; I would generally assume the one with the Masters is a better developer.
- Given two developers with a similar amount of work experience, one with a Masters and one with a Bachelors; I would generally assume the one with the Masters is a better developer.
- Given two developers, one with a Masters and no work experience, one with a Bachelors a couple years of work experience; I wouldn't make any assumptions about their skill (they're certainly different, but I wouldn't assume one is a better developer than the other).
The same logic holds true for me when people try to convince me that a degree in CSci offers no benefit at all, and that it's better to just self study and start working. There's nothing you can learn during a degree that _can't_ be learned by self study; but there's a lot of things that are much more likely to be learned in an academic environment.
Most institutions have been granting CS diplomas to people that cannot code for decades.
Those people tend not to find long-lasting programming jobs.
But research scientists aren't necessarily great engineers, and engineers aren't necessarily great research scientists. That's OK though, it's why we have separate disciplines.
After graduating most CS grads seem to forget foundational CS stuff anyways.
I never got a CS degree. So I trained myself. While working and also a lot of a year when I was unemployed during the .com crash.
Which only made me more frustrated to find that most jobs are just "you nail these two boards together" and most programmers are clueless about things like what the "relation" in "relational DBMS" means.
In construction - in the UK - there are first and second fix "chippies". First fix do the big stuff and second fix do everything after the plaster is dry(ish).
There are also cabinet makers and furniture makers, turners and several more wood related trades. Here, woodworker is a synonym for carpenter and carpenter is rarely used as a job title. Generally a specialisation is used.
I happen to own an IT firm and I prefer to see engineering qualifications on CVs (resumes). I'm a rather shabby Civ. Eng myself! I graduated just before the shit hit the fan in the UK in 1991 - recession, >10% unemployment (in Devon and Cornwall) etc. Can't recall what inflation was up to at the time but I think it was worse than now.
... So I ended up in IT
But I'd say about 20% of the students that graduated still didn't understand things like references very well. They'd write their code with "foo.bar" and when it didn't compile they'd randomly change "." to "->" or add random asterisks around until it compiled.
(Classes were primarily in C++. We did some classes Java as well, and in my 3rd year Microsoft came in with a big sponsorship to change all the Java classes to C# classes.)
A CS degree is worth nothing. A masters is worth even less.
I recognize that it's kind of gatekeeping but it feels like more and more diploma mills are just hanging out degrees, not to mention all the coding camps out there.
Becoming licensed is mostly needed if you want to work on your own. You pay less taxes but have ZERO safety, so in theory you should earn much more and work for many customers. But those shitty consultancy companies hire consultants and pay them very little.
Isn't it a bit leaky to be called that?
I've given a couple hundred technical interviews over the last 10 years. This has been the case for a long, long time—even from fancy colleges.
Software engineering experience can be gained at a code school, or by working on open source projects, if you already know some coding from one of the many free online resources.
Many would argue that this is already happening. Moreover, we have no plan to stop it, so it's no good saying that we should not lose progress toward that goal.
Hard interview questions are asked for the same reason hard questions are asked in SAT tests. The assumption is that if you can do a hard thing in the interview (implement a mutex using a semaphore or whatever) then you can probably write simple code too.
A few years ago I was working somewhere that had a low hiring bar. We were using an opensource graphql library to interact with some service. The graphql library didn’t expose some queries we needed, so I forked the library and added those queries.
After I left I heard through the grapevine that they were very unhappy with my work. They didn’t have anyone on the team who was skilled enough to maintain the code I wrote, so I left them in a bad place.
In retrospect I’m still not sure what else I could have done to solve the team’s graphql problems. Not that, but what?
Having said that, I know great, intelligent programmers, sone of who worked on compilers, who are simply bad in a whiteboard setting. Having a filter is necessary, I’m just not convinced a Google-style interview filters exactly the right thing.
The low hiring bar is relative. If all you do is put together crud apps and it's mostly stitching things together you might not even need a formal CS education. In fact a lot of people I know are wildly successful because they found their niche. They would have an exceptional bad time in a whiteboard setting.
What I don't know (I can't try the 'free' Copilot without providing a credit card), is how well it understands unusual constraints and separate pieces of C++ classes that make the real programs we write.
For instance, a simple test would be: "write a fibonacci function but skip the number 5". Does it do the right thing?
Another challenge would be a C++ class with several fields. That's what our real programs look like: composite data objects, mixing several different algorithms together to implement complex behavior.
Maybe my data structure has a hash table of items, as well as a direct link to the largest item. When I say: "write the function to insert a new item in the list, and remember to update the largest item if it is larger than the current one", would Copilot do the right thing? Each step is easy in itself (adding an element to a hash, comparing an item to another one).
So starting with 1 and 1, we get 2. 2 and 1 is three. 3 and 2 is five. What do we do from here? Do I not add 5 and 3? If not, then what do I add? 3 and 2 again? Do I repeat 3 and get 6? What does it mean to “skip 5”?
// Print the fibonnaci sequence, except the number 5.
int main() {
int a = 0;
int b = 1;
int c = 0;
while (c < 100) {
c = a + b;
a = b;
b = c;
if (c != 5) {
cout << c << endl;
}
}
return 0;In general it won't solve all your problems, but it's helpful for automating simple things like this (but you still need to test edge cases). With this prompt in Python (which I'm more familiar with):
from dataclasses import dataclass
from typing import TypeVar
T = TypeVar('T')
@dataclass
class MaxDict:
items: dict[T, float]
max_value_item: T
def add_item(
def add_item(self, item: T, value: float):
if value > self.items[self.max_value_item]:
self.max_value_item = item
self.items[item] = value
I think the hypothesis is, rather, that people who can successfully run that gauntlet are smart and motivated. The details of the problems they’re solving along the way are almost incidental.
I'm coding audio DSP along the lines (real example here) of, "Given that the Dolby noise reduction system is deeply part of the sound of classic seventies and eighties recording, what parts of it can be generalized to make DSP processing producing the same general effects on the sound, and having done so, how should this be tweaked and adjusted to optimize for pleasurable results?"
This doesn't exist on Copilot. Damn near nothing I do for work exists on Copilot, and even if it (being an MIT-licensed public code library) gets slurped up into Copilot, Copilot has absolutely no way to determine whether a 0.8273 in the algorithm should be leaning towards 0.83 to brighten the sound of something happening, that it would do this, or why you'd want brighter rather than darker, or whether you are right in wanting this or would do better to go darker knowing that other things will happen to the listening experience.
Copilot's inability to do these things is much like Dall-E's inability to do art. There is much visualization and a striking absence of purpose or intention. With Copilot, it will do all the boring or common stuff for you. If that's your job, maybe you're not aiming at the strengths of humans, rather you are aiming at things too easy to automate…
The way that job interviews continue to fixate on having a huge amount of information stored in your head, rather than your ability to be resourceful, is so frustrating. Oh well, maybe it's about time I go back to Leetcode and see how quickly I can grind out a recursive DFS algorithm...
Or the world will shudder at the Byzantine monters we can pump out day after day
Actually Copilot can really help people like you.
Even if the mechanics is the easy part, knowing how to touch type and knowing an editor like vim, can help you get your thoughts from your mind into code faster.
Copilot can do the same thing. It can act as a super powerful text editor augmentor to get your thoughts into code faster.
Just because the straight ways of a racing course aren’t the most challenging, doesn’t mean that going faster on them won’t improve your racing performance.
I think that’s exactly the intended audience. Copilot just cannot write code for me, but it’s really good at inferring patterns and giving me the next step.
All the boring repetitive coding has disappeared.
This is why I care more that you can tell me about the algorithm at a high level, and reasons to use it. I don't care about you writing code outside of a computer. I don't write code on paper and then write it in my editor. I open up my browser, look up the documentation and refresh my understanding of the algorithm to make sure I am not overlooking anything. I do some research to figure out if I even need it, maybe there's some even superior way to do what I'm trying to do in my programming language at that moment.
This is the thing that you won't be learning about in a bootcamp. Coding is easy. Knowing what to code is hard, sometimes very hard. As is keeping it simple.
In both cases I think this is a real opportunity; we can let students get more quickly to bigger problems and systems thinking by leveraging these tools. It requires professors to start thinking innovatively about how to teach and assess these subjects.
You have to be careful with the output it produces, as it often looks sensible but is wrong. Especially if it contains numeric constants. But on the whole it speeds things up, the same way good code auto complete does.
There is nothing wrong with the student assemblings and fixing parts of code that exist. Doing that requires understanding those parts, how they interacts and what you can do with them. It's not killing tests. Plus as professionals, we all do that anyway. Heck, I learned doing that. I'm paid doing that.
However, there is something wrong with academic exercices that I see in the wild. Things that are very abstracts, requiring way more than programming knowledge, or stuff that are asked out of context completely. Students are both understimulated, and given too hard bites to chew. It's a terrible way to learn, and to be tested IMO.
Tests nowaday don't tests for knowledge, they test either for compliance, or act as a filter instead of a feedback loop. This is not education.
In fact, I've yet to see school IT course that I didn't find deeply flawed, and I completely understand the students for cheating at them to that they can suffer as little as possible. They are not given a chance to prove themself or progress, they are not respected. They are forced fed junk and asked to spit it back all shiny.
I remember when I was a student, and I went to 11 different schools because of my twisted life. They all sucked. And I say that as I spent half my student life at the top of my class grade ladder.
I regret having cheated so rarely now.
There was only one professor and course that I still remember 20 years later: CS408, Software Engineering, Professor Wortman, UofToronto.
Class project was in four phases, cumulative (basic functionality for the application in phase 1, progressively additional functionality in other phases, frequently strongly interacting with previous phase code).
Here's the kicker: After each phase, you had to swap your code with another team. So you had to pick up somebody else's code, figure it out, and then build up on it.
Few of us that had real-world working experience loved the course and flourished in it. This is what we are training for! This is what programming is like! You are taking real code and building a real thing with it!
About 250 other students signed a petition to the Dean on how this is unfair and awful and they will not put up with it. They were just too used to / spoiled by 16 years of 5 assignments per semester with abstract, entirely separate questions of 1.a), 1.b), 1.c), etc.
All I could think of - if you did not like this course, you are about to not enjoy next 4-5 decades of your life :D
Other than this one course, I can say that I'm a prolific, enthusiastic, life-long learner, and my university experience was the absolute dumps - it was far less about learning, and far more about bureaucratic hoops and artificial constraints and restrictions. I was top of the class in some hard courses (generative compilers etc), mid-pack in the some of meh courses, but in retrospective, my life opened up when I was done with academia and could work and learn in the 'real world'.
Genius idea! Though if my professors did this, I may have been scared away from a career in programming
The first issue is that it's really hell on the TAs with all of the administrative work needed to do the swapping between phases. Software engineering is already one of the more annoying classes to TA, and this makes it even less palatable.
The bigger issue is that, on the basis of fairness, you grade in later phases is very heavily dependent on factors that are outside of your control. If you get someone's code who just didn't complete the assignment--that is probably going to happen once a semester--you are going to be at a disadvantage. Even if there is an option to appeal to the TAs and get the unacceptable code base replaced with an acceptable one, that's still likely to be a few days' worth of work lost learning the codebase, discovering its incompleteness, and now you have to start over again.
I'll also point out that it's not exactly real-world experience. It is extremely rare that someone will be dumped on a codebase without a prior author remaining on to help them ramp up on the code.
what? I must be living a rare life.
The only times I can think of where the previous author was able to help was either during onboarding (where you are expected to be given smaller-scope tasks with the context of that code being predefined for you) or during work on certain specific singular projects as a team.
But at any large big tech company, you are probably gonna be jumping around more than a few different codebases (more often than not, large and being worked on for years), even if you work on a singular team/product. And most of the time, you are expected to figure as much as possible on your own, only reaching out to the people when you get stuck or encounter a specific issue. Other than that, you are expected to be able to build at least a janky proof of concept with as little help as possible.
Most of the time, the help I sharply needed was rather targeted and not really related to the code itself overall. It would be something that is kind of extremely difficult to guess on your own without it being explicitly told (aside from docs/code comments), things like "everything looks fine, but I get stuck at this auth step, what's wrong there?" - "oh yeah, you gotta auth as a part of this specific group, so you need to be added to this security group in this config file".
I am not trying to say that i am some genius and can get the structure and overall mental model of how a codebase works all on my own easily, only with the specific "gotchas" (like the ones mentioned in a paragraph above) giving me issues. Not at all. The only way I can mentally get a structure of the codebase and how the whole system works and operates (even as a rough/simplified model) is by looking at the design docs/documentation (very helpful, but far from sufficient on its own, more like a supplementary material of variable quality) and, most importantly, making small code changes and debugging the code using breakpoints to see how it all flows and where. Imo, understanding code through debugging and reading a lot into the code is an irreplaceable step for understanding a section of any significantly-sized codebase.
For the fairness aspect, I would also let students pick any Stage N-1 that is not their previous N-1. That would also teach em that not all is created equal (and probably identify approaches / ideas that they can incorporate in their work)
Seemed trivial at the beginning of semester,right?
...did I mention my team had previous real life experience?
We put extrnsive code and documentation and FAQs online (this was like 99 or 2001 don't remember, but we'll before Github etc) , and committed to 7 day support of our code base. We got 18 out of 22 picks after phase 1.
Of course competition was much stiffer further phases, but that too was like 3 valuable real world lessons learned for everybody!
I got an A. By being humble reflective and proactive, showing the instructor that I was looking for solutions to the problem and giving an appropriate amount of time and effort to the class.
Now for pedagogical purposes it might be that the professor chooses the best solution or a substantially worse solution.
There might be some concern that this unfairly advantages one group over the others but it’s not completely clear to me that it does.
The way this is more likely to work out IRL is that the last person who actually understood the design - not just the what's, but the why's - had long since moved on, and the "prior author" that you have access to can, at best, tell you where the dragons are.
1. As noted, each team got to review and pick the code they would take after each phase as long as it's not your own. I remember drawing a graph of picked code lines and it was fascinating, evolution / survival of fittest at work. But it wasn't something one didn't have control of.
2. I don't know why it would make TA work either better or worse, we may not be fully understanding each other. You have 22 programs to put through a standardized test suite based on formal requirements all teams had to code to. It was far easier to grade than any other assignment I ever had.
3. I guess different people have different real world experience. All of COTS/ERP and a lot of enterprise applications are somebody else's code. Millions of lines of somebody else's code. So that's the norm in my life. YMMV :)
This is hilarious and is a good example of how sw engineering works in the real world. Love it.
I'd expect that most students would just "rewrite" the code they received from the other team and replace it with their own.
That behaviour by the students disgusts me.
It seems to be a natural part of aging, to start to complain about education and kids these days because it's not the same as when we were in school.
I try not to do that, because I never forgot how my parents generation said the exact same about us when we were in school, and I guarantee my grandparents generation complained when my parents were in school.
The most effective policy might not be to try and mechanically prevent cheating, but to explain that, this is information that you're going to need if you actually want to go into the field and if you're using copilot here you are damaging yourself in the long run.
Maybe it's not fair to give as good grades to students who are cheating on assignments as those who are actually doing the assignments, but at the end of the day is the purpose of the course to measure the student or to teach the student?
That has never, ever stopped anyone motivated to cheat. Even here on highly-educated HN, you get posters with open disdain for what they learned in college.
> "... at the end of the day is the purpose of the course to measure the student or to teach the student?"
Both. Otherwise, the college diploma really does become the meaningless scrap of sheepskin that its detractors claim it is.
Because of this, there will always be considerable pressure to detect cheating and remove offending parties from the program, be that the students copying answers from the internet, or the institutions failing to detect the problem before handing out a degree.
I feel we'd probably get the same value to society if we just made grades 9-12 optional. It would remove the people that don't want to be there, improve the quality of instruction, and college would actually be a meaningful achievement so we no longer have requirements for entry level jobs to require a Master's degree.
I don't teach CS, but I teach an online course in a STEM field. I give out a similar message ("you're only hurting yourself in the long run").
My anecdotal evidence is that my warning does nothing--I have trap questions on various quizzes/exams and the frequency of cheating hasn't changed with or without the warning.
My suspicion is that the students who cheat feel this way: "This is just some bullshit hoop that I have to jump through, so it's okay if I cheat. I'll figure out the important stuff when the time comes."
I mean, they're delusional, but I understand the mindset for cheating.
edit: amusing story.
Faculty members in my department are required to take an on-line course on how to handle hazardous waste. There are a few hours worth of videos to watch and a test that you need to pass at the end.
Last year, one of the faculty members took the test, compiled the answers and emailed them around to everyone else (to save them the time for this "bullshit task.").
I said to him, "Isn't this precisely the kind of shit that makes every faculty member angry when the students do it?"
That being said, it was a bullshit test.
Parental (other similar) pressure can indeed be quite strong. Absent of that pressure they wouldn't be there in the first place, so perhaps it is not so much being delusional as a rational response to their environment? If they cheat their way through the parents are happy and then they can return to the life they otherwise would have lived. The time lost is unfortunate in some respects, but at the same time if you're cheating, the time investment likely isn't that great and an acceptable cost given the circumstances.
> the kind of shit that makes every faculty member angry
I find it curious that the customer using the product in an unintended way would be upsetting to the vendor. What drives such emotions? In my businesses, I couldn't care less how the customer uses my product. If they're happy, I'm happy.
It is possible for the customer to also be the product. Especially in the age of salable data, that is becoming more and more common. However, when that is the case there is incentives given to the customers to shape them into what selling them as a product requires. In context here, that would mean something like giving discounts to those who don't cheat, which I am not familiar with any college doing, so... That brings us back to why would a vendor get emotional about the customer not using the product as intended?
Because your customer, in so doing, is stealing from your other customers and undermining your reputation as a vendor.
Plus, you're disheartened because this interaction is a waste of time, regardless of the fact you get paid: if money was all you cared about, you wouldn't be in education.
That's always going to be the case, though. Imagine you built software that helped businesses find customers. Using it the intended way they can realize a small number of new customer each day distributed across your customer base. Now, someone finds a new way to use your software in a manner that you never envisioned and they're attracting all the customers, taking from those who used the software as expected.
But, really, who cares? You just pivot to the embrace the new way and carry on with life. That's your market now. It's fun and all to want to be the elevator operator of old, but at some point you have to realize that nobody cares about your nostalgia. Markets change.
> Plus, you're disheartened because this interaction is a waste of time
But it is not. The students are only there to abade social pressures and cheating their way through gives them what they want out of the deal. Very few students care about the academics.
> if money was all you cared about, you wouldn't be in education.
If you don't care about money, why are you marketing the social need so hard? This is like a car manufacturer marketing that their cars are great gateway vehicles when committing crimes and then lamenting that criminals are using their vehicles to get away...
Before the ridiculous, albeit successful, "If you don't go to college you will end up flipping burgers" marketing campaign, college only attracted a small number of students who were serious about learning and all was well with the world. Colleges still spend an inordinate amount of time justifying why the cost is worthwhile to keep up the social need illusion.
It's not hard to revert to the natural state. Fact of the matter is that you (not you personally, perhaps, but the group) don't want to.
And yes it's a waste of time! If you want a fake credential then it's much cheaper and easier to just lie on your CV. It's not like many employers will check your certificates, at least not in software. I'd really rather people did that than waste my time first trying to teach them and later handling their academic misconduct cases.
If I'm marketing social need, this is the first time I've heard of it ;-)
It is, but an acceptable cost to keep the social pressures at bay. There are a lot of things in life that are ultimately wastes of time, but worthwhile to maintain a civil society. Such is life. If you are happy to escape to the deep woods away from all others where every moment of your time is purposeful, good on you.
> If you want a fake credential then it's much cheaper and easier to just lie on your CV.
What would that accomplish? The business world couldn't care less about your past achievements. They don't care that you performed in a play, rode a horse, swam in the ocean, or went to college. There is no value in even putting a legitimate degree accomplishment on your CV, let alone faking it. If anything, it detracts from your standing as it shows that you're so useless you couldn't come up with anything more relevant to share about yourself.
Instead, it is parental pressure that puts one into these schools when they otherwise shouldn't be. And those parents will turn up at your graduation. Once you've thrown your cap in the air, so to speak, nobody will ever think about it ever again. Getting to that one moment in time is the so called hoops that one is cheating for.
There's a lot more going on here than maybe you realize.
First, the students are not your only customers. The administration is also one of your customers. If you make the course TOO difficult and fail everyone, your "administration customer" will find a way to get rid of you.
Conversely, if you make the course too easy and everyone gets to cheat and everyone gets an "A", then many students who pass your class may not pass various professional certification exams that are more rigorously controlled (i.e., no cheating). If that happens, then the school may lose accreditation for programs that you are associated with (i.e., if all the nursing majors fail their certification exam, your nursing program may get nuked). Then you get nuked.
Third, whether you like it or not, your relationships with your students is something you have to deal with. Students who don't cheat have a tendency to get really upset if they find out you turn a blind eye to cheating. Happy cheating customers leads to unhappy honest customers.
Fourth, if your fellow faculty members feel like you're going easy on the students to get favorable reviews (or because you're lazy or because you hate confrontation), they can make life unpleasant for you, too. They also aren't too happy if they teach upper level courses and you send a bunch of garbage students their way.
If not, you might want to try a new business. You may not be cut out for the one you're in. The business world is not very forgiving, nor should it be. Those who can't adapt need to perish.
Spoken like someone who hasn't done the job and is confident in his ignorance.
Curious that a primitive emotional response has transpired instead. Given the greater context of discussion also about emotions, is there something about the job that attracts this type of behaviour?
The only remaining question is how to engage these people. I have seen good approaches to that problem and really, really terrible approaches, and it seems that most professors go for the terrible approaches. Unfortunately, engaging people who don't want to be there is a lot harder than just having a bunch of required tests.
These students are not all wrong that some courses are BS hoops to jump through. Meet them in the middle. Teach them something.
In general, they appreciate my candor.
That being said, in my experience, the students who cheat often delude themselves into thinking that they understand the material--and they really don't.*
Then they seem shocked when they take a certification exam (where they have to leave phones at the door and are closely monitored) and they bomb the exam.*
* I'm speaking in generalities here. I'm sure there are some cheaters who are being "smart" about their cheating.
Eventually, some of us came to the realization that you can never prove that somebody is cheating. People have been known to hire tutors to do their assignments for them. There is just no end.
As a result, we evaluate students on four dimensions. Programming assignments,* homework assignments, but also class discussions and group participation. Those last two count for a small, but non-trivial percentage of the grade, and are usually enough to separate and identify those who understand what they are doing, from those who are just "following along" solutions that they find on the internet.
* one addition to programming assignments includes an analysis write up. Tell me in human words what is happening. Why do you see that effect. What is the running time. And separately, comment your code to tell me how it works. Those two parts count significantly towards the grade.
Any other professors here, what have you found that works?
when i was interviewing candidates this was the fastest way to filter those who understood and those who didnt.
ask someone to talk you through how to solve some real-life problem. to look at a stack trace and describe what they see, or to run a profile and trace through the cause of a hotspot/contention. show them an issue and have them live debug to root-cause and fix. it's okay if they use the internet, SO, etc. -- that's how we all do it. see what they have to look up! just listening to the amount of depth someone can verbally communicate during novel problem solving (including additional questions they ask you) turns a 60min interview that wastes time into a 10min one that tells you if you can move on.
a favorite one of mine was to ask the candidate to describe in as much detail as possible what happens between a keystroke typed into a google search box and the search results appearing. the diversity of replies is facinating. some will tell you "google returns results from its database", others will ask you if you want them to first describe how the keyboard works at an electrical level through the USB or bluetooth driver stack.
"Never memorize something that you can look up." --someone smart
Like they picked some random port configuration issue that had stumped them for days in the past and thought cool let’s remove the internet, the command line, and ask people to solve it on the spot.
I offered plenty of plans for how I would go about debugging, but I didn’t know the one simple trick.
I agree with trying to use challenges that are closer to the real work, but it’s really hard to do that without over-testing domain knowledge.
For instance, I ask candidates to do some asynchronous control flow. These are all candidates with js listed as their best language and I offer to let them look things up or to show them the promise apis they need, but a certain percent just refuse to engage with the problem because they don’t have the domain knowledge and seem to feel they’re being tested unfairly.
The problem does a really good job of showing a candidates grasp of all the tricky parts of js, so I keep using it despite the drawbacks.
i dont mean that the problem is only verbally communicated. i mean that the debugging process the candidate does is verbally communicated.
in the scenario you're describing you'd be sitting at that machine with access to the internet on another machine. like, you know, in real life.
> The problem does a really good job of showing a candidates grasp of all the tricky parts of js, so I keep using it despite the drawbacks.
async stuff in js is pretty good, but it's also easy to go too deep on it with some bizzarely poorly architected code. many js devs still fail to grasp all the implications of closures, or how to avoid memory leaks, or how to work with the GC rather than against it.
It's a shame our mass education systems cannot use it. They simply cannot apply such a humane method to hundreds or thousands of students. They are reduced to applying their bullshit test questions because it's the best they can come up with at their scale.
I really, really hated those parts of school, second only to group projects. I do appreciate why teachers want that, and I hear you about how it can surface those who are cheating, but I do wonder if it's causing the class to suffer for the alleged misdeeds of a few
No person an island.
And if they are, I don't want to be on a work team with them.
For me at least, the dynamics of "School Group Project" are basically completely different and separate from "Work team". I happen to do well in both, but I do not enjoy both: the motivations and structure and dynamics and goals and timelines of "School Group Project" are so much more artificial and ultimately pointless, and obviously to all from the start.
Same things with discussions - I happen to be an engaged student, usually front row, hand always up, discussing with instructor and team and colleagues and everybody. I like to be engaged and figure things out together. But I'll never be half the developer than my colleague who barely speaks a word unless asked. He's friendly, meek, polite, and excellent team player and developer - he just does not initiate conversations, especially in group settings. I can imagine he'd have a nice big 0 in that category if that was a grade criteria.
Which is not to criticize the professors who try to use group discussions; just to point out, this is not a solved problem at scale any more than interviewing/hiring. There's too much humanity and too few absolutes :)
In school it never bothered me having freeloaders get the same grade for my work, so long as they stay out of my way and don't "help." There's a very famous mechanic's poster: "repair: $5, if you watch: $20, if you help: $1000"
Honestly I think moving away from grades is going to be the way forward. Part of this is also making university, etc cheaper so you’re not financially incentivized to not take a class one, or multiple more times.
A lot of people hate how tech interviews are done, right? Cause they have nothing to do with the job the majority of the time. Those are places where I can see improvements.
However, some jobs do genuinely need you to interact with a diverse set of people all the time, for those you need to make sure for the sake of your company & the candidate that they match the criteria. Other jobs, you need only maintain strong relations with small groups, and the requirements for those are a lot easier to hit - even the most introverted people can do well in smaller groups that don't change a lot.
I do really think a lot of evaluations that happen during education though makes a confrontational relationship with learning. You're constantly judged about your _ability to learn_and that has far reaching impacts beyond your first twoish decades of your life. You get scared of making mistakes, you get scared of taking risks, etc.
So, evaluate when it's important for the function & safety of the job, and the candidate. Skip evaluation when its evaluation for the sake of evaluation.
There's real life consequences to this kind of evaluation. Huge consequences. Students cannot afford to make mistakes. There's huge pressure and anxiety before and during a test because the stakes are so high. Failing at this stuff can cost someone their future: future jobs, future career, even the student loans that enable them to study in the first place may only be afforded to students who get good grades.
Is it any wonder people cheat?
Isn't that the whole point of all this? You can't call yourself a med doctor if you have no idea about being one, if you don't know about the key knowledge areas.
Therefore you're not a doctor and don't get to have that title.
It's about filtering out those who can't
I'd rather people had better opportunities in life that didn't depend on education. There's a lot of people who hate school and are just going through the motions because they believe it's the only path to success in life.
How is that going to stop cheating? Surely that will just make people more inclined to cheat, since they have less chance of solving it on their own.
Cheating on programming assignments has been rampant forever at every undergrad institution I have experience with and somewhat present among graduate students. My experience spans about 25 years in that space now.
When I taught an introductory class, I gave open-book exams with no laptops or phones allowed. About 1/3 of the class was unable to write a syntactically correct for loop in Python despite our textbook being an introductory Python-based book chock full of examples. It was pretty clear that a subset of students were either working together on project assignments or out-and-out having someone else do the assignments for them. I mainly compensated for this by having a large part of the grade being a 1-1 meeting with me to talk me through the code. That and the exam had the effect of actually making cheating somewhat less worthwhile. But this approach simply doesn't scale these days - my max class was around 32-33 students and the time I spent meeting with students was insane. I haven't taught in a few years and understand that class sizes of 150+ are not uncommon. I could have never used the same approach with that many students. I probably would have doubled-down on exams and made exam length darn near impossible to finish without actually knowing the material well enough to do without referring to the textbook.
I was teaching more from a practice-based viewpoint so mostly I came up with "weird" projects that mirrored problems I spend time on (data cleaning, using existing libraries to do neat little things, and also having students pick a personal project to implement that I helped them scope appropriately).
We bitch and moan about interview whiteboarding but given grade inflation its kind of hard to trust university credentials. Grade inflation was kind of disheartening in that the worst students didn't really get a poor grade. But I also didn't have any problems with the top 1/3 of the class getting a very high grade - these students were motivated, understood the material, and often impressed me with where they got in a single semester.
He had to basically interview each student individually and ask for an explanation of the code. Most could not explain what their code was doing (because they didn't write it).
They were all given an opportunity to resubmit original work. Many could not do it, because they simply didn't understand the basics of programming, or the language they were using, or how computers even worked.
These were Masters students.
But I'm on the other side and don't have the stress of "my whole life depends on looking like an all-star" at university. Like, what do you do if you leave school with such a severe lack of actual skill?
OTOH, the last year I taught, every student in the program I was an adjunct for graduated with a job offer in hand. Maybe it all works out fine.
The other side is real world pressure for good grades and when you're young you want to party and experience life. You finished that assignment by copy pasting it. It's better than not finishing it. However you get that anywhere in life. Why is, sorry, was Stackoverflow so popular? Why is religion still a thing? People want easy answers to hard questions.
My assignments are similar to everyone else's (CLRS, JeffE, Leetcode) yet different enough that simply copy and pasting will not work.
All I ask is that they comment their code, analyze complexity and cite sources.
I've been using copilot for the last year in my live coding sessions. It saves so much time (especially in comments). If students want to use Copilot I do not see a problem. It is just another tool.
Copilot is near useless without decent programming knowledge. 80% of time it is fantastic. About 10% of time Copilot gives you subpar snippets. O(n2) instead of O(n) etc. Then 10% of time it just gives you wrong snippets.
Just like GPT-3, once you go beyond a paragraph it loses context.
As for analysis write-up: turns out, language models can also explain what the code does really well, in convincing vernacular. They generate reasonable comments.
https://twitter.com/emeryberger/status/1560618825208320001?s...
As an illustrative example, I just typed in a little test into Copilot. I only typed the first line and the text that starts with "a paragraph explaining...", and typed a "#" for each comment. Copilot filled in the rest.
def compute_sum_recursively(n):
"""A paragraph explaining what this function does and why, and explaining its running time:
This function computes the sum of the first n natural numbers.
The running time of this function is O(n).
"""
# Base case: n = 0
if n == 0:
# The sum of the first 0 natural numbers is 0
return 0
else:
# Recursive case: n > 0
# The sum of the first n natural numbers is n + the sum of the first n-1 natural numbers
return n + compute_sum_recursively(n-1)With grading like that cheating on the homework is basically pointless, so few did it.
instead of writing an article crying about it, the professor could try making some unique questions to test knowledge of the underlying concepts
Evetually I gave a copy of my code / solution to Friends Group A and also to another Group B. They didn't know each other. I said, "Be careful! If you copy, disguise it."* Deadline comes. Everyone hands in their work.
The following week, we go to lecture. Prof walks in and writes a list of names on the board. I knew each name. I knew what was happening. I waited for the shoe to drop (i.e., my name on the list). The shoe never dropped**
I believe they were all given Ds. Not sure why they didn't fail (F).
* In retrospect, this was a stupid on my part. If they knew how to alter enough it's likely they'd be able to write it themselves. That is, I all but suggested they walk on water.
** Also in retrospect, the TAs + prof had to realize I was the source. The class was big but not that big. And if only some completed the assignment correctly, the source had to be not difficult to identify. I'm not sure why I was never pulled aside and spoken to. Thank gawd.
If someone is cheating and actually expects to land a decent CS job, my response is "good luck on the leetcode questions and the 5-7 hours of questions you'll have to answer between the intro call, panels, and an offer".
And if someone magically manages to get all that way without actually learning the material they will get absolutely crushed at work. Meetings where you have to give your professional opinion will induce fits of anxiety. Making changes to your code based on feedback and not having any idea what the feedback means. Even just the amount of work alone that needs to be done at most tech places crushes great engineers. Without the knowledge it will just stack up on you even further.
The very basis of the scientific method is measurement. If you can not measure your learning progress, how are you to know if your training methods are adequate or failing?
I really don't understand this idea that testing and measuring students is a problem. Even the dreaded "teaching to the test", sounds like a good idea. If doing so would somehow exclude important learning, that just identifies an area where the testing needs to be improved.
When you say into the field, do you mean academia or industry?
Because the things copilot does for you are absolutely not the things you need to know yourself to be in industry. They're, for the most part, things we tried to put in libraries (or more ideally language standard libraries for a lot of things).
The fact that is solves a lot of interview questions just means our interview process was absolutely garbage.
And I'm skeptical of the academic side of this as well. It sounds like professors from the early 60s being annoyed that students have compilers (another tool that saves a ton of work and repetition). Y'all are forgetting that this is the ultimate lazy man's field. This isn't the first time the basics have been swept away and replaced with something easier to work with (and hopefully won't be the last).
We need some people who can do things like making those libraries. It also seems plausible that the people who have at least some of the knowledge and judgement to do that effectively will, on average, be more effective on more mundane tasks as well.
Larry Wall is one who wrote (somewhat drolly) about the virtues of laziness, but there was nothing lazy about what he did.
Do we? For things as simple as copilot tends to put out? Why?
Do you also believe we need to keep people around that do other automated things? Plowing fields by hand? Hand compiling higher level languages (as the first lisp compiler was bootstrapped)?
I mean, keep the information around. Don't go burning textbooks on subjects just because we automated something. But, what exactly is the value proposition of having students do these things?
> It also seems plausible that the people who have at least some of the knowledge and judgement to do that effectively will, on average, be more effective on more mundane tasks as well.
And your claim is also that this is the only way to get the requisite knowledge and judgement? Have students take in a string from stdin with a format that changes every semester, munge it around, and do things with it instead.
> Larry Wall is one who wrote (somewhat drolly) about the virtues of laziness, but there was nothing lazy about what he did.
As someone who has done string munging in C, I'm not entirely convinced that creating perl isn't an effort saving defense mechanism (only half joking).
The thing is, regardless of which way you meant it, we need some people who can make the sort of libraries we need in part precisely because automation such as Copilot is no substitute (at least not yet.)
This observation does not (and is not intended to) endorse current methods of instruction or hiring; on the contrary, it supports spcebar's view that riding your way to a degree, certificate or entry-level position on the back of Copilot is not doing yourself any favors.
The point about Larry Wall is that we don't get labor-saving tools without someone making an effort.
I do see how that could be ambiguous. That's on me. I was referring to "the things Copilot does for you". Generally speaking trivial (or near trivial) algorithms.
> The thing is, regardless of which way you meant it, we need some people who can make the sort of libraries we need in part precisely because automation such as Copilot is no substitute
Fine. Some people might need to be able to implement and maintain libraries filled with generic algorithms, especially language maintainers. That's still a very different claim from the original "this is information that you're going to need if you actually want to go into the field". That claim implies that it's a universal requirement, whereas the reality is the vast majority probably don't need that knowledge.
Implementing these things is tedious, and results in a bunch of code that has to be maintained vs using something out of the standard library. For example, sort or max functions, which are at the intersection of what I see copilot generate a lot and intro to CS classes. Even without copilot, that's not really a skill that the average practitioner needs to have ready at all times, in fact I'd probably block a PR that implemented either of those things in my projects, it results in extra code that needs to be maintained and can be broken by a typo or something silly.
> The point about Larry Wall is that we don't get labor-saving tools without someone making an effort.
My comment about perl was sarcastic, and probably not helpful. That, also, is my bad.
My point is that even if this is the case for a majority of such people, we still need the people who make all the library contents that are beyond Coplilot's capabilities, and we both seem to agree that its capabilities are limited.
The thing is, a world in which a lot of people can be productive software developers, without even being capable of writing the sort of algorithms Copilot is capable of, is highly dependent on the people who design and write the libraries that implement not only those algorithms, but also a great deal else that is beyond Copilot's capabilities. The industry may not need everyone to be able to do that, but then it is entirely dependent on those who can.
One can certainly argue that there is no point in an education that stops at the ability to reproduce what Copilot does, but that would be something of a straw man, as education does not typically stop there. I agree that it makes a poor hiring benchmark, regardless of the role being filled.
I'm going to answer these individually because the answers are all different.
> a) everything Copilot is currently capable of can be found in libraries,
The vast majority, perhaps not 100%.
> b) learning how to do those things oneself is a waste of time
It's a waste of time if the pupil doesn't actually want to do it. If they do want to do it, there's probably quite a bit of value to be had. It's personal value for the student, not as a field.
Sort of like how I got quite a bit of value out of reading the old ITS documentation I found on github once, but I don't think I'd recommend it as part of the standard CS curriculum.
> c) it does not matter if people entering industry as software developers cannot do that themselves.
Correct. Assuming cannot means "without googling".
> My point is that even if this is the case for a majority of such people, we still need the people who make all the library contents that are beyond Coplilot's capabilities, and we both seem to agree that its capabilities are limited.
Sure, but that's a relatively small number of people. My point is that if using these tasks to measure aptitude is causing you problems because copilot exists, it's perfectly fine to just use something else.
> The thing is, a world in which a lot of people can be productive software developers, without even being capable of writing the sort of algorithms Copilot is capable of, is highly dependent on the people who design and write the libraries that implement not only those algorithms, but also a great deal else that is beyond Copilot's capabilities. The industry may not need everyone to be able to do that, but then it is entirely dependent on those who can.
Fair, but that's already the situation we're in with programming languages. We're entirely dependent on those as a field, and the vast majority of practitioners wouldn't be able to create a compiler or be anywhere near competent in language design. I would put the large frameworks (e.g. spring) in the same category.
These trivial algorithms become like the opcodes for a particular processor. Someone has to know them, but basically everybody not working on a compiler can ignore it (unless it particularly tickles your fancy).
I do have one small issue with your wording, however. Specifically the use of the word "capable" in this bit:
> without even being capable of writing the sort of algorithms Copilot is capable of
The people going into the field definitely need to be capable of implementing these sorts of algorithms. You're going to fail at so much of software if you aren't capable of something that trivial. It just doesn't need to be taught. These algorithms can be looked up if they're ever required. The ones that come up frequently will be naturally memorized and the others won't. This is a field where you have to learn new things, often without any sort of available expert in the subject, likely this will happen with an entire language or how to use a particular library.
I'm comfortable with copilot the same way I am comfortable with a calculator. Sure, if you have to, you should be able to do a logarithm with a slide rule, but I'm ok with not teaching university students how to use a slide rule. And if a professor assigned a bunch of homework under the assumption that students were going to use a sliderule, and the students all used calculators, I'd tell them to just drop that particular lesson.
In your last paragraph, you consider the case of calculator vs. slide rule, but one can, I think, make a more apposite case with calculator vs. learning arithmetic (and even if you don't agree that it is more apposite, the point can still be made!) By your logic, we should not be assigning any arithmetic problems that can be solved with a calculator.
But why stop there? Even current calculators can do much more than basic arithmetic, and their capabilities pale in comparison to those of general-purpose computers. I do not believe you can teach mathematics in a way that begins at a point beyond what has been automated, and before long that may be true for computing as well. This has not led to the demise of mathematics education, and neither will it for computing, for the same reason in both cases: anyone who needs to use these tools to get through the introductory classes will wash out later - which brings us right back to spcebar's original comment about only hurting themselves in the long run.
I think that's an excellent point, and one whose nuances shouldn't be un-examined.
I guess the difference to me is that arithmetic feels more fundamental. I may be using fundamental in a weird way here, but I don't mean it in the sense of something that you build on. I mean it in the sense that arithmetic itself is the idea that's trying to be taught. Whereas a sorting algorithm has both a explanation as a semi-platonic ideal and as an implementation and they're very different things. You can fairly effectively wield the implementation only using the idea in your head, mostly regardless of your understanding of the implementation. In fact, it's generally considered good practice to hide an implementation from your eventual users.
I'm not so convinced that you could do the same with arithmetic, although I'm open to arguments to the contrary.
Now that I'm thinking about it, I think this is the dividing line between math and computing in my head. The ability to separate out the real world part from the ideal and operate the former using only the latter.
Perhaps I'm mistaken, but I would be very surprised if a mathematician could operate without understanding arithmetic, but I've seen quite competent programmers construct APIs just fine without having been taught sorting algorithms.
They may lose in the long run, but the non cheaters aren't even allowed to run
What it will do, however, is make it difficult for CS departments to evaluate students based on homework, and it's not clear to me that this is such a bad thing. My experience with university CS courses has been that assignments are largely auto-graded, with systems akin to unit tests. In the common case, an overworked TA quickly spot-checks the source code to look for obvious signs of academic dishonesty, but that's it. So, universities are left with the following choice: give everyone an A, or grade differently. In the optimistic case, this might even produce a grading strategy that improves the value of a CS degree as a predictor of programming competence. It likely won't, in which case we have a status quo.
[0] This is a problem in its own right. I recently referred a grad student as a job candidate in my current company. I'm the most senior backend engineer, and this grad student was a highly competent contributor to my open-source project, which currently plays a strategic role in our backend. Said grad student is unusually bright and productive in a complex specialty (distributed systems), and I had to push our VP Eng very hard to hire him. His initial reaction was along the lines of "he doesn't have any industry experience, so we don't know if he's good". It gives me nausea to think about how many outstanding, 2-sigma engineers are rejected because companies don't know how to evaluate their talents. In the end, we ended up hiring him, but only because I stated that I would take full responsibility for his productivity. I regret nothing. </rant>
College is the new high-school: everyone goes, only to get a job, because you have to go to college and get a CS degree to get a job. Why? A lot of "compute science" isn't used in software development at all.
I think we should replace colleges with "boot camps" which are unpaid internships which teach high schoolers how to work in actual software development, in various fields (let's call them "entry level jobs which don't require 5 years of experience"). Why unpaid? Because companies will love it, and it will still be better than colleges where you pay a full year's salary in tuition.
What about people who actually want to pursue higher-ed computer science? Just don't replace all colleges. It seems like most people getting into CS are not doing it for academia, those that are will have a better time taking courses with others who are passionate about the subject instead of simply using it to get a high-paying job.
How does looking at job experience help to distinguish good from bad developers?
My sense is that hiring managers think academic experience is so far removed from the needs of industry that that won’t be able to understand or evaluate the merit of academic experience. I’m of the opinion that they’re mistaken, but ah well…
I don't think this is generally true, especially for algorithms and smaller programs: Googling 'fibonacci python' takes 3 seconds.
Seems roughly the same amount of effort, and since Copilot is not free anymore most students won't bother paying for access on any kind of scale that matters. Whoever was going to cheat will cheat regardless.
Besides it's one of those "you won't have a calculator with you everyday" fallacies. If you can solve problems with Copilot in class you can also do it at your job later on.
Copilot is free for students: https://docs.github.com/en/billing/managing-billing-for-gith...
Most book algorithms are in pseudocode so you need to know how to write a loop or a function in the language you are using.
And just the act of typing it line-by-line will focus attention and help cement concepts in the brain (assuming one really wants to learn).
Hitting TAB in Copilot doesn't do any of those things.
> We compare the hybrid semantic ML code completion of 10k+ Googlers (over three months across eight programming languages) to a control group and see a 6% reduction in coding iteration time (time between builds and tests) and a 7% reduction in context switches (i.e., leaving the IDE) when exposed to single-line ML completion. These results demonstrate that the combination of ML and SEs can improve developer productivity. Currently, 3% of new code (measured in characters) is now generated from accepting ML completion suggestions.
So 3% of all code at Google is now written by AI autocomplete, and Google developers who use this technology are 6% more efficient than those who don't[0].
Is it bad that students are cheating on homework with this? As bad as it was when students used Stackoverflow to cheat, I reckon. But will working with AI models be an essential skill to learn for new (and existing) programmers? About as much as getting comfortable with Stackoverflow was, I reckon.
[0]: https://ai.googleblog.com/2022/07/ml-enhanced-code-completio...
There might be AI written code in the future and surely the "no code" shills will latch onto it with religious fervor. But, much like "no code will eat the world", this too will fail. Until general intelligence is capable of producing large scale systems that need to be coddled and orchestrated, bespoke perhaps even brand new approaches to business problems, etc, there will always be many highly paid humans there.
The conspiracy theorist in me makes me think that all of this push to shill this product constantly is yet another way the FAANGS are attempting to dilute the pool enough to lower wages. Engineering is highly skilled, highly technical work. The actual work of someone higher than junior level typically evolved way beyond simple CRUD coding exercises. When you spend more time making sure copilot meets standards, the code is correct, etc it's simply not that useful. It feels much more like the fancy test case generators for some IDEs. For very trivial, borderline rudimentary cases, it works wonderfully and saves time. Once things become complicated and messy it just gets in the way.
It is also not clear to me what the control group was, neither how they were selected nor what they did instead of using the ML model.
Some of the autocomplete was okay in that it saved me some keystrokes at times, but in general, it felt like more of a distraction. I especially hated the suggested comments; these were almost never close to what I intended to write, and it would always give me a pause when they'd pop up, and often forget what I was about to write.
There were a few instances where it magically did autocomplete a surprising bit of code, and I enjoyed the novelty when it would happen, but that wore off quick.
You need to recognize what it's good for, like generating code similar to what you've written but a little bit different.
For example : I had a project in Rust where I would print a sentence describing an error that occured. I had a big match block with my possible error variants. The error enum was declared in the same file.
And so, when I added a new error variant, I just had to place my cursor at the end of the match and watch copilot generate exactly or almost what I needed, a similar block of code that the other cases, but with a different message and using the fields of this varient intelligently.
In the same vein, just have them write code out on paper during exams and coding competitions like the way we did it back in my day. It's not as if they're not going to need that skill anyway when they get whiteboarded during interview loops when even the most lenient employers get exasperated at "CoPilot-only" graduates.
</tongue-in-cheek>
Maybe more assignments like:
- "Here is a function that is supposed to generate a fibonacci sequence, but it is not correct, explain why"
- Here is a sort function. What is the name of the algorithm implemented by this sort function?
The reality of any career in CS is going to involve a lot of maintenance/understanding of existing code any systems. Copilot's failure to replace all human programmers isn't because of it's inability to cough up complex code, it's because it can't communicate with the product owner, debug a program amidst a time-sensitive incident, or explain its work.
And to some degree we will! I certainly can't do long division anymore, and I struggle to navigate my own neighborhood without GPS. This is a bit sad, but I think something of an inevitability.
If history is any guide, academics should embrace the new tech and try to teach students bigger things that copilot can't yet handle, with the assumption that they'll use copilot to fill in some details--the same way we now teach students to use graphing calculators to solve problems that would have been out of reach for their grandparents.
There are so many things I never learned as an undergrad because we were too focused on algorithms. Database design, testing strategies, different architectures, programming language design, etc. Maybe these things can come to the forefront of the curriculum if we don't need to drill basic algorithms.
These are not the same thing at all. Yes calculators and such have largely made long division pointless, but it's still quite useful being able to get around without GPS. especially around construction, GPS is awful.
You need to know the limits of the tools you're using. If you just say "I'm always going to use GPS, because it's always better" you are really doing yourself a disservice.
My friend who copied my Pascal programming assignment is now a director at a $1B hedge fund. I'm the founder of a series of scrappy, ramen noodle startups. I guess it's the "Senator Blutarsky" effect:
3 years later, the education ministry decided that was BS and let everyone bring calculators to math exams. Surprisingly, math scores didn't go up much.
A calculator actually helped me understand physics a lot better, especially things like slopes.
I use Copilot every day and I can assure you it makes a lot of mistakes. I think that, at least in the short term, CS teachers will still find assignments where it makes mistakes.
You really win as a professor when you design a project that naturally causes you to reach for the knowledge you’re trying to teach. One of the best was my networking class where the project had us secretly rediscover tcp by trying to invent reliable transmission over UDP and then once we sufficiently cut our teeth the second half of the course was implementing a real working version on top of the professor provided userspace ip stack.
To be honest there should be a class called language model querying because for simple tasks it does well but eventually you will have to debug the output (not just copilot but GPT3 and dall-e).
There's an AI for that!
Do you think the skills taught in that class would be relevant for very long?
That's why getting an Internship is always a good idea.
# I solved this problem by using the % operator to get each digit from the number. I declared and initialized a sum variable to keep track of the sum of the digits of the number. I set a temp variable equal to the input number. I then used a while loop to loop through each digit of the number. In the loop, I used the % operator to get the remainder of temp divided by 10. This gave me the last digit of the number. I added this digit to the sum variable. I then divided temp by 10 to remove the last digit of the number. I repeated these steps until all of the digits of the number were added to the sum variable. Finally, I printed the value of the sum variable.Then you were able to compile and run code locally on your machine. Then the IDEs came along, with syntax highlighting and then linting.
Copilot is just the next evolution in writing code, and saving you the effort. I bet they railed against Timsort and it's inclusion in the Python standard library, but now who honestly writes their own sorting algorithms?
This kind of generational gap reminds me so much of the "Kids are spending too much time reading" > "Kids are spending too much time watching TV" > "Kids are spending too much time on their phones" rants.
The fibonacci example doesn't make much sense either due to how trivial it is. Without copilot you are just copying from the definition of the function and with it copilot copies it for you. The other algorithms that follow are both online and probably in their textbook.
1. Start a new python file with the line "# This is an implementation of an HTTP server"
2. Press autocomplete a bunch of times and stop somewhere when "enough" libraries are imported
3. Press "#" and let it autocomplete the comment about what comes next (global variables for PORT and BUFFER_SIZE in my case)
4. Again, Press "#" and let it autocomplete a bunch of comments and functions ("get_file_extension", "get_file_content_type", "get_file_size" in my case)
5. At this step, I had to cheat a bit since it was generating too many "get_*" functions, so I started a comment with "# Initialize" and let it autocomplete again to get "init_server_socket"
6. From here on, it generated handle_request, parse_request, get_file_path and send_response automatically.
7. Lastly, I wrote "def main" and let it autocomplete again.
This produced an HTTP server which almost worked. I just had to fix a small issue with "file_path", since it expected that files were stored in the root directory, but I wanted it to load files from the local directory.
The code is not great since it does not handle most errors gracefully and is vulnerable to directory traversal, but I didn't even have to think about the HTTP protocol while writing it, so it is still quite impressive.
A few good points:
It generated a huge selection of MIME types which I probably would have had to look up by hand.
The generated server is multi-threaded!
It automatically reuses the socket, so I do not have to wait a minute every time I restart the server.
Here is the final code: https://bpa.st/CTRA
I leave it disabled more often than it's enabled, unless I'm doing something simple like wanting to copy a file, which in Go is stupidly obtuse.
It also does a fantastic job of filling in method parameters from my local variables, again not 100% of the time, but 90% of the time it nails it.
Lest anyone think I’m saying don’t teach important concepts or that such things no longer matter: there’s a whole world of principles and fundamentals that can be taught and assessed without fizzbuzzing 100 students who already had access to Google and StackOverflow before Copilot came along.
Yes, directly assessing hands-on coding would be much better, but these tiny snippets aren’t really indicative of real-world tasks that will be needed anyway (given that almost anything so foundational will almost certainly be in a standard library), it doesn’t really prepare students for real-world development work, and there’s a general expectation that new graduates are going to be very very green and therefore need a lot of handholding regardless. Even more than that, you’ve no way of detecting plagiarism already in these assignments given that a couple of basic rename refactors is going to stop you from proving the code was lifted from elsewhere to begin with.
If assignments that are easily automated are a waste of time, do you have any suggestions on how instructors should help students move from "I know literally nothing about writing code" to "I know the basics well enough to solve problems that aren't trivially generated by Copilot"? In my anecdotal experience (albeit years ago) as a graduate assistant, for students who are totally new to programming that phase often lasts throughout the typical intro to CS course and it's not unusual for it to stretch into the second level course.
It's an improvement of our toolbox, students SHOULD learn to use it.
It's not much worse than using c instead of assembler, or java instead of c, or ...
Throughout time people have invented new tools that makes their predecessors more efficient in their proffesion.
Too many programmers these days behaves like the workers did when automatic looms were invented. Or when the chainsaw was invented.. or the computers.. or the...
And yet, here we are, learning reading and writing like it's 500BC.
The unavoidable question is, will our and their kids be able to function without their machine overlords?
In the 80s I had to come up with my own solutions because no internet and no books, being 11.
With the internet I searched it with the error messages. Pre 2000 I got useful and accurate results, post 2k I got more and opinions than results, which had to do with the amount of mediocre intellect on the net and Google trying to provide a larger list of results, but mostly the stupid made it onto the net.
Om github you already have the stupid I'd imagine since it's such a popular platform, although many have left it when Microsoft tool over. But many also let their work be abused by Microsoft to sell the Copilot service, so I'm guessing the stupid is very present on Github.
The evil of taking someone's work, that they shared with good intent to help other people, run it through a machine network and sell it to the "other man" again.
And now it's even corrupting our children and our children's children.
Pure evil Microsoft.
I absolutely disagree with how Copilot has been trained on the intellectual property of others, and I think Microsoft/GitHub should be taken to court over it (and I say that as an IP cynic, but if you play that game, you should stick by the rules), but this technological cat is out of the bag.
Students (and everyone else) should be thinking much harder about how they test and verify the behaviour of any given piece of code and how they design systems to stick together. This class of technology is going to automate away many of the jobs where you just pump out code. This is fine. It’s no different to someone inventing a burger-flipping machine, and I’m sure many here would agree it’s better if humans didn’t have to flip burgers for a living (modulo solving employment being necessary for a decent/tolerable existence). It may turn out that the next step in software productivity is not a new generation of cool and highly-abstract declarative programming languages (4GL) but simply automating away the drudgery of writing code in the extant kinda advanced languages (3GL) which are ‘good enough’.
Arguably Copilot doesn’t change much about the software development process. You can already get software developed pretty cheap if you farm it out to low-quality contractors or a lot of juniors straight out of school. There’s a whole branch of the industry that will hire anyone straight out of university to manage and front development teams in South Asia. But as we say, quantity has a quality all of its own. And the only way to manage software (not code) quality with this method is to verify and test that the deliverables actually meet your criteria and expectations.
It’s currently a great time to focus on TDD/BDD and code specification because our time spent on the D is about to get a lot shorter. There is still a place for artisanal hand-crafted code. But the mechanical loom has arrived.
More important than even fundamentals, though, is modeling a successful software development mindset and problem solving technique. Too many professors have not _had_ any success in software development, which is sad. I can count dozens of examples of my own personal growth that could have been handled in college, that I instead ended up inflicting on my first employer.
Whatever the current state of the art and tools of the trade are, that's what they are, and probably the bulk of courses will just adapt to reflect that state of reality and incorporate that into the courses.
Outside of maybe some specialist courses that may still exist but most students wouldn't need to take, it would just be expected that most students use the current tools to accomplish the tasks. Even going so far as to provide free student access to any paid tools to avoid giving the rich kids an unfair advantage.
The nature of posed problems, and their teaching purpose (why they are posed and what you get out of them) would just change from what they are now.
Whatever the "calculator" can do, simply becomes uninteresting and not required for most people to worry about.
My first couple years of school I was taking mechanical engineering, and it was right during a transition where we were doing both manual and cad drafting. There is a lot of geometry tricks both physical drawing and math required to generate accurate views of any shaped objects intersecting with any other shaped objects, viewed from any angle, sliced at any depth.
We spent a lot of time and effort on that. That is all done by magic now inside the cad engine.
Someone still needs to know it, but 99% of people whos job is to design or otherwise manipulate models of objects do not need to know it.
I, who did learn that stuff, don't think an engineer who didn't have to know how to generate an accurate view with nothing but a pencil and a calculator is a less capable less insightful less quality engineer.
If anything they may be better for not having to spend as much of their intellect on mechanics & implementation vs ideas. The knowledge & skills I'm talking about don't contribute to their understanding of the real job, they aren't foundations that the later stages build on. It's just labor of no value at all except to the cad engine designer. It had a value in the past because it was a form of literacy. It was simply the only way to document the ideas you were fabricating.
Perhaps copilot is like that.
I don't know and I'm not saying it is, but it could be.
I haven't used copilot yet, so I don't know how intelligent it is at producing solutions that it hasn't seen before. Assuming it's not that intelligent, one solution would be to just add small, tricky variations or constraints to well-known programming problems. So instead of just asking "find the sum of numbers", ask "find the sum of foo numbers", where foo is some made-up property, like a number contains even number of odd digits. Then require the implementation to count down instead of counting up.
But on the other hand, I found copilot useful as a tutor. When somebody asked about an API, I didn't have to akwardly search the documentation, I fired up VS code and have copilot suggest something (without the students seeing it of course ;-)).
My wife taught CS, she did catch cheaters pre-Copilot, and my first thought it that she probably would enter test questions and print out a reference sheet for Copilot generated results.
I'll note that defeating plagiarism detectors is easier than many people think. I and one of my students wrote a paper on an automatic technique to defeat plagiarism detectors, and it was highly effective: "Mossad: Defeating Software Plagiarism Detection", Devore-McDonald and Berger, OOPSLA 2020 (talk and paper link here: https://2020.splashcon.org/details/splash-2020-oopsla/14/Mos...).
> techniques inspired by genetic programming with domain-specific knowledge to effectively undermine plagiarism detectors.
My creaky memory of CS Theory makes me suspect that, ultimately, Mr. Turing's halting problem may make resistance to cheating futile. [1] It's interesting that the paper is in OOPSLA, I used to follow the work from it more, especially when I listened to IEEE's Software Engineering Radio.
Thanks for your work in this area and your reply. DieHard and DieHarder are funny, creative names for fault tolerant memory managers. It's also nice to see a fellow humorist. At least we can rest assured that cheaters get what they deserve, like my classmate at Columbia. [2]
Regards.
[1] > He, for one, welcomes our new AI overlords.
Yes, let's suggest a Borg-like partnership with them. I'd be willing to drive "self-driving-car" tow truck--if they haven't invented a self-driving one yet.
virtually every cheating student has always slightly altered their solutions. At best copilot saves you the five minutes of Googling and refactoring. For the committed cheater who wants to avoid a week's worth of homework and learning that's hardly meaningful. And in fact probably not even that because you still need to actually read what copilot outputs because it makes obvious bot-like mistakes quite often.
There's an emerging divide between those with experience with AI tools and people who tried them a few times and got hyped and blogged a maximalist view of consequences.
For lower level courses we had online assignments, and pen and paper exams - which is where the majority of the course grade came from. If you didn't [intimately] know what you were doing on the assignments, you'd fail the classes due to the tests.
Not everyone who take CS1* courses are CS (or similar eng) majors. We keep crying "Everyone should learn to code" and now that's made easier than ever some of continue to complain. Huh?
Full disclosure: I started to use CoPilot a couple of months ago. It does not remove me from the process. It does allow me to shift more brain cycles from the mundane to the more difficult. My general feeling is, I have yet to get close to taking full advantage of what it can offer me.
That being said, I think Microsoft should be fined $100B every time they give one of their products away for free to students. Hopefully that will be sufficient incentive for them to stop so being so overtly anti-competitive.
I remember reading about a framework that actually sort of does that - create randomized programming languages. It was meant as a security feature against code injections iirc, but technically it should work as a solution to OP's problem.
Although I guess some of the other suggestions are better and easier to implement - ask students to explain their code (select one person randomly per problem, it's quite common anyway) and give more weight to tests.
Ultimate end user for systems like these are not humans.
The perceived productivity boost that people get from it could be interpreted as a condemnation of how verbose a lot of coding still is even in known areas. Lots of brain cycles wasted on doing things suboptimally that we can already do optimally.
Just make sure Copilot is explicitly against the rules, there's not much more you can do than that
If one cares about cheaters (and I'm not convinced we should), doesn't that solve the problem?
Not giving anyone ideas or anything, but if you feed it your previous assignments and it learns your writing style it should be hard to detect with minor editing.
I guess Microsoft will know which students did their assignments on their own when they apply for a job at Microsoft.
Does it integrate with Lazarus?
So it might be more like bringing a tennis racket to a grenade fight.
Although he was an older physics teacher who only really did computer class ("Informatik") because there was nobody else available or (more) qualified to do the job. When later on there were classes on programming, he made us use TurboPascal instead of something less cluttered and actually useful in modern economy like Python.
That same teacher later on performed some show experiments about induction and electromagnetic forces in physics class where he let a magnet fall through a glass tube with several copper coils around it. So far so good, but he then tried to readjust one of the coils one the glass tube, which predictably shattered in his hands, nearly severing some fingers and actually severing some tendons, which IIRC were later replaced with other tendons from his legs or something. Needless to say, not the sharpest tool in the shed.
My other physics teacher during 10th-12th grade physics classes was a severely esoteric nutjob, preferring to hold long rants about "science" being able to diagnose diseases in people by somehow shining a laser at a drop of their blood and interpreting the reflection/refraction because there was a "connection" between the person and their blood (or something like that, I honestly tuned out his ravings after a while). Apparently his wife is/was some kind of homeopathic "healer".
All that is to say, computer and physics class teachers for me were "very fun" and "useful", since one did not have a clue that their students were "cheating" and both were not very in tune with modern and/or scientific methods.
This is another clear indicator that most students today see college as a higher class trade school, not as a place of higher learning.
College may have been a place of higher learning at some time before the last thirty years but I doubt it.
Javascript may be the only one of those four worth teaching as it has different patterns and different ways of solving the same problem.
> Here’s an approach that’ll work for sure: use some, let’s call them alternative, programming languages that Copilot doesn’t really know. (...) Sadly, I have news: Copilot’s love for programming languages knows no bounds! Racket! Haskell! ML! (...) Copilot is a ravenous beast: if any code in any language found its way into a GitHub repo, it’s already swallowed it up and is hungry for more, nom nom nom.
Not sure how true this is in practice - I only used Copilot once, with Python - but if it is that might invalidate this concept
A reader of my article sent me this note, pointing me to some relevant hot-off-the-presses work (reproduced with permission):
-----
A group recently evaluated the performance of Codex (and another model) on 18 programming languages:
https://nuprl.github.io/MultiPL-E/
The high-order bits are:
1. The latest and greatest Codex is now twice as good on its own benchmark suite than the original version published a year ago.
2. It's just as good on Python as it is on JS, Scala, C++, Swift, TypeScript... and other languages are not too far behind. It's not bad at bash of all things.
Paper here: https://arxiv.org/abs/2208.08227
Seems like they would have had to solve the “how do I know what you know you know issue”.
For the classes that are about the law (contracts, torts, criminal procedure, etc) it pretty much all comes down to exams. I only had one take-home exam [1]. All the others were in-class.
The exam questions were mostly essay questions. There would be a couple paragraphs or so describing some situation, and you would be asked what the legal outcome should be. Your essay needed to identify the legal issues involved, cite the relevant cases and/or statutes, and argue how those applied to the facts of the case to support the result you were trying to argue.
[1] Note: I am not a lawyer. I went to law school when I got burned out with programming. By the end of law school I was no longer burned out, and decided I'd rather be a programmer who knows a good bit about law than a lawyer who knows a good bit about programming.
Someone who can explain some code understood it (or memorized the explanations)
from an academic point of view, it is not that bad. even for the people in the "memorized" category, they memorized the explanation... brings them quite close to understanding IMO
https://theconversation.com/i-no-longer-grade-my-students-wo...
Good practice for white boarding interviews.
You can test loop-writing in an oral-exam, too. Thus your proposed solution for GPT-3 writing ethics essays is so good, that it fixes the original problem.
Sorry, what's the point of that? I don't think it would make any difference for cheating?
The blue book is a standard. https://www.amazon.com/BookFactory-Exam-Blue-Ruled-Format/dp...
https://www.thoughtco.com/blue-book-1856928
In the classes that I had that used blue books for exams, the exam book was free and given out by the proctor.
E(very) S(ingle) W(ord) S(hould) be (no [ACCEPTED] completion here) W(ritten) W(ith) C(onscience). You should be responsible for your code submitted. Are your feeling confident?
Won't touch Co-Pilot. Probably you are spending equal time proof reading than _understanding what needs to be written_ in advance. At least that is what I am hoping about. Of course you don't have to proof read, and most things can probably be grasp in a glance. But you rely on a black-box. From a corporation. To me that sounds like agreeing with a dictator.
Wouldn't promise my craftmanship for things I didn't grasp in advance. But maybe that's a culture hindrance. I like it though. I want to know the code I am corrected about. I want to archive mastery.
Just pose better questions and more open ended assignments.