Using Google’s code history to write more code
ai.googleblog.com
ai.googleblog.com
But they are typically lazy and copy and paste that code from existing places. If you can remove the last copy from the code, they will finally stop.
And now that special idiot is a piece of software that never sleeps, never eats, and never forgets? What could go wrong?
The wisdom of crowds, if it applies to teams, is in communication. I talk you out of cutting corners, and sometimes you talk me out of it. There's no communication in a training corpus. You just get the average of all of the code that wasn't so horrible that people stopped what they were doing to fix it. Which is to say, half of the code is barely tolerable.
Google's monorepo and tooling for changes across the entire repo make it possible and common to fix entire classes of mistakes at once. Bad patterns lingering in copy/paste is not an issue in practice due to a combination of such changes and linter presubmits that identify and flag any recurrence in the future.
>more concise ways to render the logic
This isn't better either.
Edit before the shitstorm:
This isn't necessarily better either.
But I also don't want to read Haiku either. All the burden is on the reader, and there are many readers.
Usually what I want is a short story. Long enough to be accurate, short enough to lose distracting and unnecessary information.
It's much easier to get one model to change its behavior, than 1000 software engineers.
What I’ll buy: Using LLMs to write less code (maybe even delete a lot)
Without thorough automated tests I wouldn't trust myself to be able to effectively review the code.
People write a lot of code because the alternatives usually have their own set of problems. Compact/clever code is hard to understand, debug and evolve. "Write only code" is only half a joke.
Languages like APL and K have some appeal due to their expressiveness but they are understandably rejected by most engineers because it's really hard to build a sustainable large scale system with many engineers
Pain in information and they throw away a lot. It's a form of job security, and not a particularly benign kind.
Our job is to replace painful processes with code. If you can't do that you aren't fulfilling the basic premise of your profession.
I find people creating these DRY abstractions with a lot of knobs to be strange. End up with dryWithRain, drySpicy, dryStrict, dryTurbo
then the abstraction is a maze of confusion
1. The change history of a repository contains more information than just the final state of the code. The change log constitutes useful training data which can allow a model to learn things it might not get from just being trained on completed code.
2. Software engineering consists of a lot more than just writing code. They've been able to train a model to do other tasks, such as reviewing code and responding to review comments.
3. Operations such as renaming a variable can be modeled as a single high-level action, rather than as a set of raw code edits. A model trained to generate such higher-level actions can potentially achieve better results. "For example, a rename might touch a file in dozens of places, but a model can predict a single rename action."
4. A developer's recent activity can be used to help predict next steps. They give the example that if you add a parameter to a function and then move the cursor into the function documentation, the model can predict that your next step is to document the new parameter. "Without history, the model wouldn’t know whether the missing function parameter is intentional (because the developer is in the process of a longer edit to remove it) or accidental (in which case the model should re-add it to fix the problem)."
A longer example:
> ...we started with a blank file and asked the model to successively predict what edits would come next until it had written a full code file. The astonishing part is that the model developed code in a step-by-step way that would seem natural to a developer: It started by first creating a fully working skeleton with imports, flags, and a basic main function. It then incrementally added new functionality, like reading from a file and writing results, and added functionality to filter out some lines based on a user-provided regular expression, which required changes across the file, like adding new flags.
This ability to emulate a natural sequence of authorship steps to create a new code file seems similar to "reason step by step", but specialized for coding.
And given how software development historically has scaled sub-linearly with org size, I expect we will see interesting developments as these tools continue to improve. Advancements should magnify the leverage differential.
However, if tools like those discussed in the article can change the equation of how software development scales with org size, then maybe that changes things.
I do agree that large orgs will gain a lot here as well. But I’m not sold that they will gain more than skilled individuals, proportionally.
In large software companies (I've worked for a few) the bureaucracy always seems to scale exponentially more than the 'productivity'. So you end up waiting on a build system that takes 5 hours, you have to use some old library because it's pinned in a monorepo, reviews from other engineers that take days, a Staff Engineer wants to give his opinion so the project gets delayed another week, another manager feels threatened by your idea so he shuts you down, etc. AI won't solve any of this, it will just allow increasingly more convoluted bureaucracy to be created.
Compare that to a scrappy developer or startup, they can run literally run circles around big tech. The only thing is they don't have is as much money or resources, but they have a ton of advantages getting things to market, where AI can be used to increase productivity without all the red tape.
- Available to Enterprise customers? No.
- Can other researcher replicate their result? No.
- Is this a recruitment blog? No.
I have no idea what is the target audience for this post.
I'll admit -- it's intriguing to me.
This is quite a stretch of an acronym, and makes it feel like they came up with the acronym first and filled out the meaning later...
People really underestimate how divergent the tech is in there. It's really a whole parallel tech ecosystem that's been evolving independently for 20 years.
And I'm not aware of anything outside that ecosystem that has anything close to the level of instrumentation and integration of the entire dev process.
Hopefully in 6 months there will be a generally-available tool for this that isn't too expensive. The dream is written tasks -> code, with the ability to interrupt and tweak. Copilot and ChatGPT are "yeah, but..." for coding - they do some of the heavy lifting but don't get all the way there.
IMO, people who are poo-pooing this are threatened, and have too much of their ego wrapped up in being good at code. Code is a tool to serve a purpose, most of the meaning is at the social or product level. Let the computer win the game!
Me: Save -> run
LLM in background: commit "added function foo that does X"
Even for when my code doesn't work, squashing commits is way easier than unsquashing commits.
Well there is bard that is available in many locations.
...and if you want APIs there is a plethora of them on GCP.
If you would like to use some 2002 technology by all means adopt Hadoop.
Do you also complain about e.g. research papers that are published on systems that you can't use?
I think it’s a VERY good thing that they’re sharing this knowledge to the people instead of being stored internally. You know there will be someone who will take inspiration from this to create software.
Do they have no intention to make a product out of it? This is not something Microsoft would've done. They directly announced Copilot as a product from the get go instead of parading their tech.
Users and employees can swap places on case to case basis
Please explain.
It’s been slow going but I do think we will see a cultural shift towards releasing more products backed by research like this.
If something is not reproducible nor public, the researchers could claim anything they want, as long as it somehow sounds plausible.
A researcher working in a corporation may have good reasons to exaggerate results or cover the defects:
- individual prestige (higher salary / better job)
- impact on company valuation
- potentially being able to claim a patent if someone else actually manage to implement the real deal
- etc
Academics on the other hand need to maintain strong relationships with industry to be in a position other than just chasing grant money (via publications). They need their paper to be accepted, or they’re in trouble. They certainly care about prestige and patents just as much as any corporate researcher.
Code is a liability, not an asset.
The benefit that some code (maybe) provides is an asset. The code as such is strictly not.
Nobody would even look at your code if you don't compensate them with some benefits.
Exactly that's why nobody touches random "free code" even with a ten foot pole if there aren't very strong reasons to do so.
If you could have the benefits without the code any sane person would prefer that.
Code is just cost. You don't want code as such as already just owning it has steadily increasing costs. You just want the benefits out of it.
If there's a solutions to a problem where writing new code can be avoided this is almost in any case the cheapest and most preferable solution.
Only junior devs think code has value. But the truth is you get more value if you're senior skilled and smart enough to avoid wasting your time on writing (more) code.
Code is a means to an end, not a means in itself!
Given that, the best approximate measure of technical dept (and with it security issues, bugs, and in the end of the day cost) is: Lines of code!
https://www.jamesshore.com/v2/blog/2008/an-approximate-measu...
Now, a smart and really helpful AI for programmers would try very hard to achieve the stated goals but delete on it's way as much code as possible. Because that's the only way to lower cost in the long run and keep things maintainable.
Any AI that does the opposite, and just creates mostly completely mindless piles of more and more code, is like an oven heated by dollar bills…
I really don't get how something such obvious isn't seen by so many people. Especially people that have to handle all the pains created by more an more code on a daily basis.
1. Hire a talented developer
2. Store all their development interactions over the years
3. Train a model on these interactions
4. Fire the developer
5. Do this for thousands of top-end developers
6. Start assembling virtual AI teams trained on these developers, see which teams perform the best
7. ... something insane is going to happen
Has google started layoffs yet?