4,714 karma · joined July 19, 2024
side: https://hcker.news
ryan [at] hcker [dot] news
> OpenAI appears to have gazumped Tristan Buckmaster (pictured) and Levent Alpöge, a duo of mathematicians labouring on the task.
What's hilarious is that the economist has gobbled up Levent's disingenuous "this was just a personal project" narrative.But see, here I am back at the edge of the rabbit hole, and you're trying to pull me in. I refuse!
If you do end up training a model, send me an email and maybe I can tie it into the site.
1. I apply a domain and keyword filter to the feed
2. The content of the remaining articles are run twice daily through a Modern Bert-based classifier fine-tuned to detect AI-related content (~8000 training examples)[0].
It also filters out Github repos that contain AI authorship. My backend scans: - Commit messages for agent attribution
- The contributor graph for agents
- Repo files for instructions/configs
RSS feed here: https://hcker.news/feeds/atom?period=day&ai=excludeHN Frontpage minus AI: https://hcker.news/?view=frontpage&ai=exclude
You may also like Small Web HN: https://hcker.news/?view=frontpage&smallweb=include
0: The original workflow for this was a little different. For the better part of the past year, I had an AI agent detect AI-related stories and raise ~20 stories to me to make a judgement call on. After a while, a workflow like this just doesn't make sense when small models can do it equally as well. The training data is based on the machine and my labeling.
https://chromewebstore.google.com/detail/hckernews-helper/gm...
https://addons.mozilla.org/en-US/firefox/addon/hcker-news/ https://hcker.news/notes/browser-extension
(Safari ext coming soon)
The extension itself follows the principle of least privilege. It doesn't even have access to the HN site cookie.
A model that was released a couple months ago scores 50% higher than SWE-2, a model released today, on an out-of-sample benchmark. Can I say I’ve come out of this more impressed with Sol?
Like you said, TB2 is saturated. Nobody would bat an eyelash at 90%. And yet here comes SWE-2 coming off top rope with an emphatic 92.4%. this is the definition of bench maxxing.
Terminal Bench 4 was released a couple weeks ago, so the difference you're seeing between the two scores can be interpreted as "how well does this model generalize to new problems"? More crudely: "how benchmaxxed is this model?"
People need to understand how all these AI company policies around training data work before working with them, because it seems that people have no clue. Some things you should internalize:
1. Opt your data out of training with the AI companies. There are multiple reasons why this isnt an airtight solution (see the following)
2. Never press the feedback button. Once you do, your entire conversation will get slurped up, retained, and used in training data. This is especially important with coding agents because they can sometimes be too trigger-happy with a root directory find command, which can expose a *ton* of your personal data without you even knowing.
3. Understand ai lab-specific policies. For instance, Anthropic / Claude Code has data opt-outs, but commits to keeping (for 7 years) and training on any of your chats that trigger their safety classifiers, even if they're false positives! Anyone remotely familiar with CC over the years understands how easy it is to trigger their safety classifiers.
4. Providers of open models will not be any more charitable with the use of your data than the large US labs. For some reason, I've noticed here that people have a fairly loose security/IP posture around open-model providers because "I'm not doing anything important." It's very difficult to properly judge the importance of your data, and whether or not it can or will be used against you. The best posture is to always be more paranoid than less.
Another post that made it on the front page presented as fact that OpenAI "stole" the proof from Thom. There's no excuse to use one's own ignorance as a reason to fan the flames of anger towards AI companies. Like, we need to pump the brakes here because things are getting unnecessarily nasty, and it's not hard to imagine a mentally unwell person who sees stuff like this feeling motivated to do bad things.If it is found that OpenAI and other labs are not respecting the training opt out then, I agree, there is reason to raise a commotion. But, with Thom and Buckmaster, accusations of malice are more better explained by incompetence (naivete).
edit: i'm sure i'm going to be accused of being some bot shill of the AI labs again but, people, this stuff all falls under the umbrella of common sense opsec.
For all intents and purposes, "low" is pretty much the same as turning reasoning off, and "high" is similar to "max". "High/max" performs way too much reasoning, takes forever, and causes costs to balloon. They need a proper "medium" setting.
I get it that they're probably focused on pushing performance right now, but the ergonomics of the model aren't great.
OpenAI is no stranger to rivalry with Anthropic but 1. it's not like user data is sitting around on some kitchen table somewhere and 2. I consider OpenAI to be as economically motivated as any other actor in this space and playing around with user data like that would destroy their business.
There are things that Buckmaster alleged and things that he speculated. The entire training data thing is speculation. If this is pissing you off, then you ought to evaluate how you ingest information.
Seeing as how xai, anthropic, and openai are all having issues at the same time, it has to be a common provider, right? Maybe cloudflare?
edit: ChatGPT and Grok are back up. Claude is not. Typical.
edit: actually, you cannot even log into your OpenAI developer account. Something's wrong.
If AI forces people to re-weight their priorities (i.e. be less on the internet), perhaps that's a good thing?