sdesol

With humans we have a decent understanding of what they are capable of. I trust a medical professional to provide me with medical advice and an engineer to provide me with engineering advice. With LLM, it can be unpredictable at times, and they can make errors in ways that you would not imagine. Take the following examples from my tool, which shows how GPT-4o and Claude 3.5 Sonnet can screw up.

In this example, GPT-4o cannot tell that GitHub is spelled correctly:

https://app.gitsense.com/?doc=6c9bada92&model=GPT-4o&samples...

In this example, Claude cannot tell that GitHub is spelled correctly:

https://app.gitsense.com/?doc=905f4a9af74c25f&model=Claude+3...

I still believe LLM is a game changer and I'm currently working on what I call a "Yes/No" tool which I believe will make trusting LLMs a lot easier (for certain things of course). The basic idea is the "Yes/No" tool will let you combine models, samples and prompts to come to a Yes or No answer.

Based on what I've seen so far, a model can easily screw up, but it is unlikely that all will screw up at the same time.

As mentioned earlier, unless you have a 6th sense for what is wrong, you won't know. If the message was "make sure to double check our response" then they get a pass, but they know people will just say "why shouldn't i just use google."

Hey thanks! The error is in the config file. Will fix this.

Yes this is a tokenization error. If you rewrite the sentence as shown below:

https://app.gitsense.com/?doc=905f4a9af74c25f&model=Claude+3...

Claude 3.5 Sonnet will now misinterpret "GitHub as "Github"

> While LLMs do plenty of awful things, people make the most incredibly stupid mistakes too

I am 100% not blaming the LLM, but rather VCs and the media for believing the VCs. Once we get over the hype and people realize there isn't a golden goose, the better off we will be. Once we accept that LLM is not perfect and that it is not what we are being sold, I believe we will find a place for it that will make a huge impact. Unfortunately for OpenAI and others, I don't believe they will play as big of a role as they would like us to believe/will.

My reason for commenting wasn't to say LLM sucks, but rather we need to get over the honeymoon phase. The fact the GPT-4o (one of the most advanced, if not the most advanced when it comes to non programming tasks) hallucinated "Github" as the input, should give us pause.

LLM has its place and it will forever change how we think about UX and other things, but we need to realize you really can't create a public facing solution without significant safe guards, if you don't want egg on your face.

In the process of developing my tool, there are only 5 models (the first 5 in my models dropdown list) that I would use as a writing aide. If you used any other model, it really is a crapshoot with how bad they can be.

> "let's only allow the LLM to do things we know it is rock-solid at."

Even this is insanely hard in my opinion. The one thing that you would assume LLM to excel at is spelling and grammar checking for the English language, but even the top model (GPT-4o) can be insanely stupid/unpredictable at times. Take the following example from my tool:

https://app.gitsense.com/?doc=6c9bada92&model=GPT-4o&samples...

5 models are asked if the sentence is correct and GPT-4o got it wrong all 5 times. It keeps complaining that GitHub is spelled like Github, when it isn't. Note, only 2 weeks ago, Claude 3.5 Sonnet did the same thing.

I do believe LLM is a game changer, but I'm not convinced it is designed to be public-facing. I see LLM as a power tool for domain experts, and you have to assume whatever it spits out may be wrong, and your process should allow for it.

Edit:

I should add that I'm convinced that not one single model will rule them all. I believe there will be 4 or 5 models that everybody will use and each will be used to challenge one another for accuracy and confidence.

If the P is silent, how can you "Prompt"?

I have no doubt the author of the press release used LLM to help them, but I'm not convinced that this was fully generated by AI. Since you got me thinking about this more, I decided to run the sentence across my tool with a new prompt that will ask the LLM to decide. Both Claude and Llama believe there is a 55% or more chance while GPT-4o and GPT-4o-mini feel it is less than 55%.

https://app.gitsense.com/?doc=381752be7fd0&prompt=Is+AI+Gene...

Edit:

I created another prompt that tries to better analyze things and they (models) all agree that it is most likely AI (+60%). The highest was gpt-4o-mini at 83%.

https://app.gitsense.com/?doc=381752be7fd0537&prompt=Is+AI+G...

I messed up and didn't really analyze the paper but the article. Since I can't edit, here is the introduction to the paper:

https://app.gitsense.com/?doc=696357b733b

which contains enough grammatical errors that I'm pretty sure it was not generated by AI.

I ran this across my AI spelling and Grammar checker at

https://app.gitsense.com/?doc=4715cf6d95689&other-models=Cla...

Note, sentences highlighted in yellow means one or more models disagree.

The sentence that makes me think this might not be AI generated is

"Researchers can use this framework to answer complex questions, find gaps in current knowledge, suggest new designs for materials, and predict how materials might behave, and link concepts that had never been connected before."

The use of "and" before "predict how materials" was obviously unnecessary and got caught by both gpt-4o and claude 3.5 sonnet and when I questioned Llama 3.5 about it, it also agreed.

For AI generated, it seems like there are too many imperfections, which makes me believe it might well be written by a human.

> Ok so just to be clear, that's an entirely different and unrelated argument from the one I responded to.

It honestly made no sense what you were saying so I didn't respond to that directly as I assumed it would be clear from my explanation as to why animals can be intelligent and LLM are not.

> LLMs can learn within their context window.

They don't learn from the context window as much as they use what is in the context window to define a probabilistic path. If you put something in the context window that it was never trained on, it would spit out BS or say it doesn't know.

> But the idea of letting an LLM write/move large swaths of code seems so incredibly irresponsible.

I do think it is kind of crazy based on what I've seen. I'm convinced LLM is a game changer but I couldn't believe how stupid it can be. Take the following example, which is a spelling and grammar checker that I wrote:

https://app.gitsense.com/?doc=f7419bfb27c8968bae&samples=5

If you click on the sentence, you can see that Claude-3.5 and GPT-4o cannot tell that GitHub is spelled correctly most of the time. It was this example that made me realize how dangerous LLM can be. The sentence is short but Claude-3.5 and GPT-4o just can't process it properly.

Having a LLM rewrite large swaths of code is crazy but I believe with proper tooling to verify and challenge changes, we can mitigate the risk.

I'm just speculating, but I believe GitHub has come to the same conclusion that I have, which is, all models can be stupid, but it is unlikely that all will be stupid at the same time.

Animals are capable of learning. LLMs can not. LLM uses weights that are defined during the training process to decide what to do next. LLM cannot self evaluate based on what it has said. You have to create a new message for it to create a new probability path.

Animals have the ability to learn and grow by themselves. LLMs are not intelligent and I don't see how they can be since they just follow the most likely path with randomness (temperature) sprinkled in.

> Most people recognize that LLM's are not currently super intelligent,

I think calling it intelligent is being extremely generous. Take a look at the following example which is a spelling and grammar checker that I wrote:

https://app.gitsense.com/?doc=f7419bfb27c89&temperature=0.50...

When the temperature is 0.5, both Claude 3.5 and GPT-4o can't properly recognize that GitHub is capitalized. You can see the responses by clicking in the sentence. Each model was asked to validate the sentence 5 times.

If the temperature is set to 0.0, most models will get it right (most of the time), but Claude 3.5 still can't see the sentence in front of it.

https://app.gitsense.com/?doc=f7419bfb27c89&temperature=0.00...

Right now, LLM is an insanely useful and powerful next word predictor, but I wouldn't call it intelligent.

> style does not scale to real software scales

I think those that dismiss AI completely will fall behind, and those that turn it into a crutch will pay for it in the years to come. I truly believe AI is game changing, as I used it to create standalone functions and get answers that saved me a day or two of research and reading. I've never worked with the cheerio library before but it answered everything I needed to know, among other things. It wasn't perfect though, as it (can't remember the model) wasted some time for me regarding the SQLite library for Node.js.

I think the issue we have right now is we are treating LLM as a final solution (mainly due to investors) instead of thinking of it as a new interface, with quirks that cannot be taken lightly. It's a bit extreme, but I think junior developers should not be allowed to use LLM. LLM is a Power Tool for developers that can easily spot BS and/or have the confidence and knowledge to fix BS that is missed.

And remember to always challenge the response with both the same and different models. No joke. Just continue the conversation for the example in the blog and ask the LLM "Do you see anything wrong with the code?" and it will spit out "Yes" and explain why.

I would have to imagine 90+ percent of people use LLM and AI to outsource their thought and most will not heed this warning. OpenAI might say "Check important info." but they know most people probably won't do a google search or visit their library to fact check things.

You do have to admit, the footer is extremely small and it's also not in the most prominent place. I think most "AI companies" probably don't go into a sales pitch saying "It's awesome, but it might be full of shit".

I do see your science fiction angle, but I think the bigger issue is the media, VCs, etc. are not clearly spelling out that we are nowhere near science fiction AI.

> 100% agree. We don't expect human developers to be perfect, why should we expect AI assistants.

I think the issue is that we are currently being sold that it is. I'm blown away by how useful AI is, and how stupid it can be at the same time. Take a look at the following example:

https://app.gitsense.com/?doc=f7419bfb27c896&highlight=&othe...

If you click on the sentence, you can see how dumb Sonnet-3.5 and GPT-4 can be. Each model was asked to spell-check and grammar-check the sentence 5 times each, and you can see that GPT-4o-mini was the only one that got this right all 5 times. The other models mostly got it comically wrong.

I believe LLM is going to change things for the better for developers, but we need to properly set expectations. I suspect this will be difficult, since a lot of VC money is being pumped into AI.

I also think a lot of mistakes can be prevented if you include in your prompt, how and why it did what it did. For example, the prompt that was used in the blog post should include "After writing the test, summarize how each rule was applied."

> I can't imagine people actually typing code by hand anymore by that time.

I can 100% imagine this. What I suspect developers will do in the future is become more proficient at deciding when to type code and when to type a prompt.

This will only work if they can ensure the product that they promote is, in fact, good. Google makes it very clear that what you are seeing is popular (or is a paid ad), but they don't endorse it. ChatGPT is seen as an assistant for many, and if they start making bad recommendations, things can go bad fast.

> Tough to tell, given nobody is turning a net profit on LLMs yet.

I suspect in the case of Meta and other big players, profit isn't necessary required to bring substantial value. Imagine their model being able to help them moderate more fairly and accurately. This alone could prevent potential legal actions from individuals, companies, and governments.

> Subsidizing open-source projects is ultimately a marketing spend

They say 2M in Vercel credits. How much is that in actual money?

I've built a code search engine in the past so I know how painful debugging long indexing jobs can be and I would certainly be reluctant to support private repos until I know I can index thousands if not tens of thousands of open source repos with no issues.

> then realize that you yourself don't have a perfect idea of what exactly this feature should be

I talked about it the last time that Copilot Workspaces reached the front page two days ago and that was, I don't think the value is in the code generation, but rather in the ability to capture our thought process. CW is currently a bottleneck in my opinion and I think the code generation will have to get pretty good before we can see the value in writing everything down vs just coding as we have always done.

I do think workspace by GitHub is a smart move. What is clear as day is, we don't have enough data to model how developers think and iterate to solve problems. With workspace, I see it as a way to capture structured data on how developers think.

This is certainly a long game though. I think GitHub with MS money can continue to lose money on Copilot for the next 5 years to gather data. For other VC ventures, I don't think they can wait that long.

Interesting. I don't think the AI code generation will live up to developer expectations, but I do see the value in "project management for developers". The value I see with workspace is not the code generation, but the ability to help developers organize their thoughts.

Can you share some of your prompts that you have used?