14,235 karma · joined May 6, 2012
It won't be about tokens in the long run, and I think the labs know that. It will be about long-term memory and user relationships, and who gets to own that data.
The disruption and upheaval has barely even started, IMHO.
Some of my favorite personal benchmarks are brainteasers with subtle changes that test whether the model 'cheats' by returning a stock answer or actually reads the whole prompt and reasons through the premises given. E.g., the usual liar/truth-teller riddle but modified with 'One guard always tells the truth, the other never lies.'
All models parrot some variation of 'Ah, yes, the classic liar/truth-teller riddle,' in their thinking traces, but only the better ones will hedge their final answer with some variation on 'The usual answer is X but if you really meant that they both tell the truth, the answer is Y.'
That doesn't mean they cheated by acknowledging that something very much like the prompt was in-distribution for them. As long as they get the job done without relying solely on training data, it's all good.
- kstenerud
Apparently so. Most people will run up the white flag after being nuked once, you see.
It was important to make them think that we had a whole stockpile of them. Otherwise, twice might not have been enough.
Also, people who are ready to surrender don't need to be nuked twice.
Where do you feel the dollar threshold should be set? At $990,000 per year you still need to listen to users, but at $1,000,000 you are basically King C of Rust Mountain and they are lucky you even let them run your code?
Should RSUs and cash signing bonuses be included in the calculation?
Mine already has been.
... as long as the reviewer model hasn't been deliberately lobotomized, that is.
(I don't disagree with the [dead] comment, either -- fortunately I tend to make small changes to code I'm very familiar with, so it's fairly easy to vet in person. The more code gets generated by the clanker, the more nervous I get, just because it's impossible to keep some complacency from creeping in.)
The solution, as usual in engineering, is negative feedback. Unfortunately, many people around here think that means "Shoot the messenger."
You won't want it back. How much time do you spend surfing through your .o or .obj files today?
That was, in retrospect, an important bellwether.
Tall women are taller than short men, so if you're hiring a librarian who needs to be able to reach books on a high shelf, you still need to look at the individual. Society as a whole suffers if you run employment ads saying "Women need not apply," so we rightfully deter that.
What does "by extension" mean? Culture is voluntary. You can (and should) walk away from a flawed culture. Race is an unsheddable fixed attribute.
Meanwhile, both culture and race see more variation within groups than they do between groups, making discrimination a losing proposition even aside from its inherent immorality.
I'm fine delegating my programming work to a clanker, even accounting, but come on. The clanker isn't going to have to deal with the consequences of booking $40,000 airplane trips to hotels full of bedbugs.
So, yeah. Back to GLM 5.3.