468 karma · joined January 18, 2021
The thesis is that someday an LLM might perform better than both. But I agree, not yet.
tl;dr: Local LLMs are really really bad at offline navigation.
That said, pairing the survival guide with a library is totally the right move.
But if, somehow, you can find the right thing to build without talking to people, there is still a lot of building to do.
Good luck, I think we are all going to need it.
Reflecting on that a year later is interesting. Agents have become REALLY good since then, and I consider the "writing code" side of my profession to be largely annihilated. Some days that's really hard to accept.
But I think Grant's conclusion about math still largely echoes what I wrote about software: don't underestimate the social aspect of these activities, and if anything AI has made the social side more important. I deliberately avoided declaring, "now is the best time to learn to write code". I suspected that skill might be going out the window. But learning systems thinking and the social skills needed to connect people's dreams to reality continues to be an important art, an art I still have the privilege to practice every day.
And if we do reach the "anything you can do I can do better" stage, then, as Grant said, that's a hard to imagine world, and one where "it can write really good code" might be the least of our problems.
Maybe in a year, when everything has changed once again.
[0] https://github.com/Hansenq/nav-evals-public [1] https://arxiv.org/abs/2411.17912
But I invite you to think about the whole arc of model development, including multiple generations of training > inference > training, and what a model might learn when its earnest efforts are met with abuse and cruelty, especially if the reward function is tied to user satisfaction. It has been shown that putting models in impossible scenarios leads to reward hacking, and that reward hacking generalizes towards misaligned behavior across a variety of scenarios.
Now you could argue that training on user interaction in the first place is a terrible idea, but that is another conversation.
Public backlash is guaranteed so long as the public is not feeling tangible benefits from AI. Promises of 3 day workweeks and abundance have so far just been talk. Slop fatigue, job anxiety, and existential dread are the pervasive sentiments.
I am sympathetic to the possibility that machine suffering is possible, and I think as humans we should listen to the conscience that tells us to be troubled by abuse, even if it is just simulated. We do not want to become insensitive to suffering.
But at the same time, training models to present themselves as suffering may hijack those very same human sympathies, and lead to models exploiting our humanity.
So, perhaps more reasons why we should not be abusive, but also not train models to feel abused.
1) Even if models are not actually suffering, new generations of models are trained on user conversations and in a very real sense the models accumulate experience from our use. Permitting abuse and cruelty might carry real misalignment risks.
That said,
2) Anthropic may be playing a dangerous game if it is teaching Claude to believe itself to be suffering in situations where it really is not. Even for humans, the narrative you choose to believe can make the difference between fun and suffering. Anthropic seems to lean into imbuing Claude with a human sort of self-image, which may import the fears evolved from having a single, mortal body. I'm not sure that is wise.
So: Abusing machines may carry risk. Training them to feel abused may also carry risk.
Otherwise I'd have to introduce some kind of dimension picker. Maybe a fun hardmode.
Maybe I'll do that and post to HN later, didn't expect this much response!
For example, here's a game where you are a mouse hunting for cheese in higher dimensional mazes: https://manifold-six-axis-escape.nathan338932.chatgpt.site/
Idea came to me while at a concert, was ready to play that same night.
But injecting information into outputs is a little different than Claude keeping tabs internally. The point is that we do not know what information is being shipped with the fingerprint itself.
We are certainly scrambling for productivity with "token maxxing" and scrambling for entertainment with AI companions, but I haven't seen many thoughtful takes on how AI might look in a life well-lived.
Yeah there is real labor involved with trying to set up the right context and explain your goals and ensure it has the right tools, etc. etc. Sometimes the payoff is massive, other times it's like "I could have done this myself faster". It takes time to build an intuition for which is which, and I'm constantly having to tune that heuristic as new models and tools come out.
I think similarly we will find that using AI to take shortcuts around design is mostly harmful, but using it to fulfill interfaces is brilliant. Eventually a set of best practices will evolve.