192 karma · joined February 2, 2023
It also has a habit of "running wild". If I say "first, verify you understand everything and then we will implement it."
Well, it DOES output its understanding of the issue. And it's pretty spot-on on the analysis of the issue. But, importantly, it did not correctly intuit my actual request: "First, explain your understanding of this issue to me so I can validate your logic. Then STOP, so I can read it and give you the go ahead to implement."
I think the main issue we are going to see with Opus 4.6 is this "running wild" phenomenon, which is step 1 of the eternal paperclip optimizer machine. So be careful, especially when using "auto accept edits"
If the model then distills that information down to a new feature, and re-examines the original prompt with the new feature embedded in an extra input tensor, then repeats this process ad-infinitum, will the language model's "prime directive" and reasoning ability be sufficient to arrive at new, verifiable and provable conjectures, outside the realm of the dataset it was trained on?
If GPT-4,5,...,n can progress in this direction, then we should all see the writing on the wall. Also, the day will come where we don't need to manually prepare an updated dataset and "kick off a new training". Self-supervised LLMs are going to be so shocking.
Open source software is a great thing for humanity, but as software engineers, we should keep the big picture in mind. Software is software. The sky has been blue and the grass has been green since long before software existed, and the sky will be blue and the grass will be green long after software ceases to exist.
I spent a LOT of money on a high-end Corsair mouse, bought the same model twice, and every time I had issues with "double clicking when should be single clicking" or "not clicking at all".
I HATE Razer for forcing you to register with your email address if you want to even customize your RGB colors. Never again. I spent time trying to write a driver for OpenRGB, but that was just too much for me at the time. And why should I be expected to do that? I wasn't even using Linux at the time, this was on Windows/MacOS!