Ask HN: How are you using LLMs or ML to improve your SWE? | Hacker News Reader

Ask HN: How are you using LLMs or ML to improve your SWE?

8 pointstriyambakam20 comments

The other day I needed to write a polyfill for a browser API. I used ChatGPT (in browser) to do it and it got me mostly there pretty quickly.

But I don't yet use any other LLM tooling like Copilot, WizardCoder etc. and I wonder what I'm missing out on.

View on HN

I treat like google. I ask it a question , unlike google I can continue to ask it follow up questions about my previous question. I find it better for complex problems then simple questions. It’s like a second brain you can outsource things too that u can normally do but would take longer .

I use ChatGPT on problems where I have really specific questions about them. For instance why I'm getting a certain return value, why a function isn't working, etc. It's a lot less headache than fumbling through Stack Overflow or the official documentation (most of the time I do check documentation first, but there are times where it doesn't have my answer). I haven't started using Github Copilot yet but plan to soon.

I don't use the "copilot" style stuff either; I am basically in constant communication with GPT-4 via the normal interface, copying and pasting stuff in/out of there and asking questions.

For a software engineer not to use LLMs now strikes me as basically "malpractice"

OK, so you've reproduced copy/paste from stackoverflow, but by using the electrical consumption and CO2 production of a small country. Congratulations.

lol no

If you think it's the same as using SO, I don't think you have used LLMs enough. For example, I have countless times just dumped errors from my terminal into GPT-4 and it gave me steps to resolve the issues. A number of times, Googling the same issues didn't provide any answers.

Can "copy/paste from stackoverflow" fix and update config files you paste into it? No.

Can it explain to you the changes it made and answer your specific questions about said changes immediately? No.

Can it give you all options for libraries in your language that do x, tell you the pros and cons, relating them to your use-case, and integrate said library into a code snippet you provide? No.

Right now, GPT-4 is tutoring me in Haskell, Elasticsearch, and AWS. Can "copy/paste from stackoverflow" do that? No.

> For example, I have countless times just dumped errors from my terminal into GPT-4 and it gave me steps to resolve the issues.

That's fine, but a pretty poor basis of comparison for what StackOverflow can do. If you identify the part that is broken (which most linters will do for you) and paste that into Google, you will get what you want roughly as often as ChatGPT furnishes a proper answer.

I disagree with the parent's pointless moralizing of the situation, but I'm also pretty disenfranchised with the state of GPT-enabled search and programming. When I ask ChatGPT to write something I've already done myself, it will often do things in roundabout or outdated ways. When you play the role of a proctor rather than a student, you can really identify it's shortcomings.

That's not to say AI-enabled coding isn't helpful. But past a certain watermark of trivial problem-solving, AI literally lacks the context to assist in certain codebases where human programmers could easily grok what's happening. Maybe my answer isn't based off the state of the art, but nothing I've tried so far has really gotten everything right. Claiming that it's "malpractice" to avoid it is a plainly inflammatory claim.

> Claiming that it's "malpractice" to avoid it is a plainly inflammatory claim

Well, I'll certainly accept the critique on that part, lol

I think that the current frontier is the completion of tasks that require large chunks of a codebase as context/knowledge.

I think one major area where it's useful is that it "knows" how to use sufficiently common libraries in your language and is simply a superior form of documentation for a lot of stuff.

For example, I wanted to do stuff with PDFs in Python, both ingesting PDFs and modifying them. For the latter functionality, I asked GPT-4 what my options were for modifying PDFs in Python in xyz manner; it gave some options, I went with ReportLab; I asked it how specifically to do what I wanted in ReportLab, telling it to fill in some half-baked function I copied and pasted in. It filled it in, gave me the imports, etc. This all took about 5-10m. There is no way it would have been efficient for me to research the original question, and no way I could have derived the same functionality from ReportLab's documentation as quickly.

I use it a lot as a replacement for "documentation" in that sense; for language-level libs, for stuff like Docker (whose docs are pretty bad). It's just a waste of time, IMO, to read people's (usually) poorly written documentation or a SO thread when the LLM can just cogently distill the information for you AND answer your specific questions about the issue.

Similarly, it's been pretty invaluable for me in learning Haskell. Syntax and other errors in Haskell provided through VSCode Haskell extension via GHC are comically bad, IMO, often amounting to a small red squiggly that's not even where the actual error is, with the message "parse error". When I have put the code snippets into GPT-4, it has identified and corrected the issue 99% of the time, and I can ask follow up questions of my own and get explanations. It's like having a senior Haskell dev at my beck and call to badger, and I think that's the general vibe of my usage.

I consider myself a pretty competent mid-level dev who can ask good questions and has a reasonable estimation of what he doesn't know, and I think that's a good Goldilocks position to make good use of the tool.

Accordingly, I think if people who are working with the same stuff in the same codebase who have themselves become relatively expert in what they need to know won't find LLMs as useful here, but if you're in a situation where you have to get familiar with a lot of new stuff, they are pretty incomparable.

> If you identify the part that is broken (which most linters will do for you) and paste that into Google, you will get what you want roughly as often as ChatGPT furnishes a proper answer.

Well most of the time, I don't need to identify the part that is broken or devote any mental energy to combing through Google search results. I just copy and paste errors into GPT-4, often without any context, and it tells me what's wrong, and I take the fix and do it, only stopping to comprehend what actually went wrong if I judge that it's worthwhile.

The UX of traditional internet search, with its multiple results of unguaranteed quality, is, I think, kind of obsolete now in many contexts. I have questions to which I want answers. I have patience for reading actual long-form content, such as good dev blog posts, books on stuff like Docker, cloud providers, "how to build an interpreter", etc. What I do not have patience for is combing a bunch of random internet Q&A stuff that is about an error that might not even be the exact error I have.

Everything LLMs produce just a statistically likely pattern based on inputs, which in this case happens to be a huge corpus of information from published information on the web. Everything it's "tutoring" you with can already be found, GPT-4 has just created a compressed summary of the most frequent patterns.

If you want to be an autodidact, seek out the information from primary sources, study it and organize it yourself. With GPT-4, you're essentially reading a sloppy cliffs notes of the topics. You'll be much better informed and be able to think more deeply about the subject matter if you adopt some ideas from Richard Feynman, wherein you try to explain what you think you know to a child, or at least someone with no knowledge of the subject. That will highlight the gaps in your understanding, and you can return to the source material to fill in the gaps.

You're right about the enshittification of search, though. Maybe ask GPT-4 for a solution to that.

Most of the time, I have not found the distillation from GPT-4 to be "sloppy".

> If you want to be an autodidact, seek out the information from primary sources, study it and organize it yourself. With GPT-4, you're essentially reading a sloppy cliffs notes of the topics. You'll be much better informed and be able to think more deeply about the subject matter if you adopt some ideas from Richard Feynman, wherein you try to explain what you think you know to a child, or at least someone with no knowledge of the subject. That will highlight the gaps in your understanding, and you can return to the source material to fill in the gaps.

The most immediate and fundamental problem with what you are suggesting is that it is infeasible. Nobody has time to learn everything they wish to at the level of depth they would ideally want. Nobody has time to read all the primary sources on which GPT-4 was trained. I find this so obvious I won't belabor the point.

I don't think any other tool can approximate the personalized "per-unit-of-time learning" afforded by a good LLM.

The topics you mentioned were picked up by others with gpt. Because they learned the pattern themselves they have a deeper knowledge then you. You might remember the prompted you used but someone else can identity the pattern and apply to other problems. You are cheating yourself out of learning.

The alternative to learning it using GPT-4 is not to learn it as deeply as other people did over much longer periods of time.

The alternative is whatever one can learn without GPT-4 in whatever amount of time is reasonable/allocated for it at the moment.

Like, take Docker. Yes, you would not be wrong to say that "by getting help from GPT-4 to do stuff, you won't learn it as well as people who didn't get that help and slogged through it over the course of 2 years at one of their last job", but this ignores that I probably will, over the course of the next several months, learn a lot about Docker in a fraction of the time it took the hypothetical other person.

I don't doubt that leaning too much on LLMs as a crutch may have negative effects, but I find this to be a surprising line of argument on a tech forum; there is such an obvious difference in efficiency between learning using an LLM versus the slog of other materials--again, aside from books, though I think they offer, ideally, a different kind of learning.

I use gpt (3.5) and it's great but deep learning only happens when you add in friction and chatGPT makes things easy so you lose that step. For one off tasks use it but for others try to solve the puzzles yourself before you peak at the answer.

It's great that you can take a different approach to achieve the same outcome. For one-off scripts with less context than answer, GPT will "work magic" as you've described.

Now, zoom out to what people actually work on. Typescript projects careening past 10k SLOC, microservice architectures with a mile-wide helm chart, and highly situational firmware programming that depends on proprietary datasheets. Unless you're an utterly disenfranchised CEO (see: Elon Musk, et. al), you will quickly realize that nobody gets paid to work on anything simple. Past a certain point, surface-level "copy and paste" answers stop being helpful. This is where I'd consider that a developer relying on ChatGPT becomes negligent; many projects force you to trade in your "let's try this snippet" instinct for investigative problem-solving. Hitting the reroll button on ChatGPT won't get you any closer to understanding the issue or preventing it from happening in the future.

I'm not entirely bearish on AI or LLMs, but if you can't see their limitations then I recommend pushing them to their limits. Ask ChatGPT something impossible (write an OS) and work your way down to more simpler requests. In my experience, every current AI model gives up long before writing a complete piece of software. Even working on smaller apps and programs (>500 lines) feels like a struggle, with current context limitations.

I think you may be misunderstanding how I and others are using it; I feel you are viewing it as very much an either-or proposition of "it either writes code for me or tells me exactly what's wrong" or it's unusable. For example, there are still questions one would have as one goes along for "investigative problem-solving" which GPT-4 is useful for answering.

> In my experience, every current AI model gives up long before writing a complete piece of software.

I don't get why you want to get it to write a complete piece of software.

Agree or disagree with it, my overall point is that once you leave the safe-harbor of simple scripting, GPT falls off hard. It's entirely possible that I've just been pushing it too hard, but for a lot of queries I get conflicting, wrong or outdated suggestions. Trying to shoehorn it in as a problem-solver in those situations has repeatedly failed on me.

> I feel you are viewing it as very much an either-or proposition of "it either writes code for me or tells me exactly what's wrong" or it's unusable.

Yeah, kinda. If ChatGPT spends it's output tokens telling you wrong, unnecessary or unhelpful information, it's not usable. Once it passes a certain bar of unreliability, it becomes useless. I'm not sure any place I've worked at in the past would feel comfortable paying me to roll those dice with ChatGPT on their systems.

I'm not against people empowering themselves by delegating AI for programming. Knock yourself out, if you want to develop like that it is entirely your prerogative.

> Syntax and other errors in Haskell provided through VSCode Haskell extension via GHC are comically bad, IMO, often amounting to a small red squiggly that's not even where the actual error is, with the message "parse error". When I have put the code snippets into GPT-4, it has identified and corrected the issue 99% of the time

Wow, GPT-4 can correct whitespace errors in whitespace-sensitive languages? That's pretty astonishing.

What kinds of things do you work on?

As a software engineer in aerospace, I have not really found it helpful. But neither has StackOverflow been as helpful to me as it clearly is to many. I think that what kinds of problems you're working on -- more specific than "software" -- likely plays into how helpful such resources can be.

The stuff I have been working on lately has consisted mostly of web apps that ingest and perform operations on PDFs and also, until recently, a fair bit of Leetcoding lol

In this context, I have used the LLMs to learn/fix/write about Docker, Google Cloud, Haskell, Nim, various Python libs and frameworks and language features, and various other sundry and misc topics.

I’m no developer I just interested in technology in general but I do like tinkering with computer and comfortable with it. I have basic knowledge in programming just a script kiddie I would say. Before LLM I would struggle to build a complete either script / app. Now I could confidently say everything I face I could ask LLM, it just like a Steve Jobs meme “there is an app for that” to “there is a minimal boilerplate to that”. Someone might said LLm bring people to the average, and for a lot of us below average that is a net positive I guess. Instead I also curious for 10x Engineer how they see the tools, how much gain they could have.

I use it to get straight answers without sorting through SEO spam and parsing pages. I can find what I need without dealing with the internet at large.

Reply on news.ycombinator.com