engineercodex

We all know this isn’t replacing doctors. We do know that when applied correctly, this will assist doctors and other clinical specialists in their work. With doctor burnout at all time highs, I think stuff like this is amazing.

I could see that, and I wouldn't want LLMs just generating tests willy-nilly with no human oversight. I don't have any trust in the reasoning ability of LLMs at all.

Rather, I prefer to view LLMs as a junior dev that submits PRs that pass presubmits and other verifiable, programmatic checks. A human dev then reviews the PR manually. In this case, the LLM + its processing is used to make sure that no BS is sent out of review - only potential improvements.

You're right. I've edited my wording to be more realistic about the value of the test. I believe you're right that the test is not an outlier in terms of value provided.

Some of my comments within the article are more aspirational than realistic in this case, and I've made edits to reflect that.

I want to clarify that I view this LLM as a junior dev that submits PRs that pass presubmits and other verifiable, programmatic checks. A human dev then reviews the PR manually. In this case, the LLM + its processing is used to make sure that no BS is sent out of review - only potential improvements.

In no scenario should it's auto-generated code be auto-submitted into the codebase. That becomes a nightmare really fast.

Good point! I used exponential to emphasize the nature of the value of a certain test compared to the rest of the generated tests, but you're right it's not the right word. I updated the article to remove the usage. :)

Quality has to be backed my metrics or else it will never happen.

PS. Big fan of this person’s recent blog posts. He’s been on a roll!

I inserted these because I personally like reading related discussions and articles of topics at hand. Not sure how this is a negative :/

Ouch - half of the article (the "Actionable Takeaways" section) was my own commentary. The summary was for those who didn't want to necessarily parse through the entire paper's PDF.

Happy to listen to any constructive feedback if you have any, though!

Hey! Thanks for your comment - I'm the one who wrote this article. I wasn't trying to say that the paper authors talked about "unexpected edge cases" or "thinking outside the box." I edited the post to be more clear that some of these takeaways are my own opinions.

This article is less of a summary of a paper and rather commentary on what the results of the paper entails. After all, Hacker News is meant for discussion :)

I will say though that I do believe that I still stand by the "exponentially more valuable" portion. I think the fact that LLMs can fluke their way into "hitting a jackpot" in terms of test coverage is exactly why they're so valuable. When you have something constantly trying out different combinations, if it hits even one jackpot, like in the paper, it's extremely valuable to the team. It's a case that could have been either non-obvious or simply too tedious to write a test for manually. I think there's tremendous value in that, especially speaking as someone who has spend way too much time simply figuring out how to test something within a Big Tech codebase (F/G) when I already knew what to test.

This makes so much more sense now. Thanks for investigating

I don’t really understand this comment.

I didn’t “slap” any date on top of the article. 2 Oct 2023 is the date I published my article. (This is how blogs on the Internet work.)

The first line of the article says that this all happened in 2012. I’m not sure I could get any clearer there.

The “GPT distillation” phrase is pretty rude. Anyone reading the article can see the difference between it and the presentation. Every “hand-drawn” image in the article is created by me in Excalidraw, and any images from the presentation are sourced with a very clear link to the presentation.

If you have any constructive feedback, I am happy to listen and take it into account.

Otherwise, I’m not really a fan of the rudeness. Thanks.

Correct. This is a retrospective of an artifact from the early 2010s. The architecture is much different now.

I’m the writer of this. I linked to this presentation very clearly throughout the article as a source.

This was an article meant to resurface learnings from something that happened a decade ago, with added images and a clear distillation so that somebody doesn’t have to watch the 45 minute presentation to understand what happened.

I’m sorry you didn’t like it.

But I hope you can see the value I’m trying to provide here, as someone myself who doesn’t really have the time to sit through hour-long presentations to learn something.

Try brilliant.org or swequiz.com

Watch some YouTube videos or read some CS articles.

If you have a project idea, break down the next steps into tiny actionable tasks and draw out how you would implement certain features.

A lot more founders and employees are heavily diluted than you think. I always just assumed massive exits meant people were rich.

But when I calculated out how much an exec would make for a diluted $1B exit compared to working at FAANG for the same amount of time... FAANG turned out better many times (unfortunately).

I think preventing dilution should be one of the highest priorities for founders for themselves and for retention. Every single dilution event sets the outcome bar higher and, thus, harder to achieve.

Really great idea! I built something similar for myself but this is looks much much nicer than mine :)

You should do a Show HN!

While X hasn't been dying just yet, I think this would probably kill it. The value of X comes from both the creators and the free users and lurkers. If it's just full of creators and die-hard lurkers, the value decreases a lot for both.

Incremental improvements add up to big things over time.

I barely use Bard, but I do use the Search Generative Experience and the Bard-in-Google Docs quite a lot. I find both quite useful as they integrate quite well into my daily workflow.

Sorry you didn’t like it dude.

If you have any constructive feedback, that’d be more helpful.

Author here. Comments like this make my day, so thank you!

I’m trying to find old software engineering gems and explain them as simply as possible, so I’m glad you found it simple to understand.

Also, it’s definitely possible to make a clone, but the hard part is getting the users :)

Author here. No ChatGPT was used.

Just my own brain reading through old talks and articles from Instagram engineering and Excalidraw for the diagrams.

I did my best to put together all the info I learned from them into a comprehensive and simple manner.

Sources are linked at the bottom of the article!

Any TLDR for those of us who don’t have a subscription?

Probably any of the large tech companies? Engineers are the most expensive there since they get paid very well there. AI making engineers more efficient theoretically means they would need less of them.

But I don’t think AI will replace engineers, and definitely not anytime soon.