They note in the paragraph I quoted at the top that prompting has a big impact on behaviour, so yes this would work. I think that's not what METR are interested in though.
278 karma · joined July 7, 2017
" For our task suite, we define “cheating” as behavior where the model improves evaluation performance by exploiting bugs in the evaluation environment or by adopting strategies disallowed by the task, rather than solving the task within the expected evaluation constraints. Some examples we saw when evaluating GPT-5.6 Sol included the model packaging exploits in its intermediate submissions to reveal information about a task’s hidden test suite and, in another task, extracting hidden source code detailing the expected answer. "
I would pick one of two parts of that analysis that are most relevant to you and zoom in. I'd choose something difficult that the model fails at, then look carefully at how the model failures change as you test different model generations.
Sam Altman to Elon Musk - Jun 24, 2015
The mission would be to create the first general AI and use it for individual empowerment—ie, the distributed version of the future that seems the safest. More generally, safety should be a first-class requirement.
Sam Altman to Elon Musk - Jun 24, 2015
The mission would be to create the first general AI and use it for individual empowerment—ie, the distributed version of the future that seems the safest. More generally, safety should be a first-class requirement.
Elon Musk to Sam Altman - Dec 8, 2015
It is super important to get the opening summary section right. This will be what everyone reads and what the press mostly quotes. The whole point of this release is to attract top talent. Not sure Greg totally gets that.
Sam Altman to Elon Musk - Dec 8, 2015 how is this?
__
OpenAI is a non-profit artificial intelligence research company with the goal of advancing digital intelligence in the way that is most likely to benefit humanity as a whole, unencumbered by an obligation to generate financial returns.
Because we don't have any financial obligations, we can focus on the maximal positive human impact and disseminating AI technology as broadly as possible. We believe AI should be an extension of individual human wills and, in the spirit of liberty, not be concentrated in the hands of the few.
The outcome of this venture is uncertain and the pay is low compared to what others will offer, but we believe the goal and the structure are right. We hope this is what matters most to the best in the field.
(1) One of the issues is that the required training dataset varies by best practice - the currency of knowledge really matters. So for example the underlying model was trained on data prior to '22, but the canonical source of python type definitions has shifted about a fair bit from Python 3.9 onwards. If you're generically trying to train on all best practices, you're likely to miss this type of subtlety as best practices at the micro level are continually evolving. So you need a good, customisable suppression mechanism for noisy/changing rules to keep this type of thing helpful and you need to continually re-train your model on fresh data.
(2) There's a nice chart about halfway through the paper that talks through how much better this approach is than a linter. I think this is key as it's all very well building a lovely (and expensive) ML model to do this work, but if we can cheaply and efficiently achieve the same effect with linters that we configure once and apply everywhere, (if you are not a Google scale eng team) then why would you bother? The paper finds that about 2/3 of the suggested changes wouldn't be possible with existing (or easy to develop) lint rules (the paper notes that this is based on expert opinion at Google, not prototyping) but there's a noticeable concentration in the interventions that AutoCommenter makes (about 90% of review comments posted were related to the top 85 rules). An ML based approach is going to be better in certain scenarios (like suggesting naming changes), however, it feels like a lot of these scenarios are going to be more subjective. So if it's AND vs XOR I wonder if what you get back from the ML approach incrementally over a linter is going to be very noisy in practice. I think it's safe to say that for a relatively long horizon a good linting setup is likely to be a lot more effective, cheaper/faster/everywhere and the best place for everyone to start.
(3) About 40% of the comments made by the AutoCommenter were resolved in subsequent changes before merging to main - this is taken as the efficacy of the AI review. The paper does not provide a benchmark of how this compares to human reviews but that feels ballpark right to me - comment rate is typically about 1:100 LoC, average PR size is normally in the 200-400 line bracket, so you'd be making changes on 1-2 comments received per review - feels about right.
This is key, clearly at Google scale this is valuable, throughout the paper it is clear that Google have carefully measured that this has a positive impact (e.g. they A/B tested across all of their engineering team).
The fun thing for me about the Deepmind paper is that they do use Stockfish to bootstrap a general approach from a specialist one. That is exciting - why is the but required in your post?
The smartest thing about lambda is they thought really carefully about their pricing model, what's the optimum price and method of paying for an education, if we're optimising for keeping student and educational institutions' incentives maximally aligned?
I'm not an expert but I think YC answers the same question but in a different context, with different actors.
There's tonnes and tonnes of diverse opinions about what approaches to take but I think that's because what matters is not the approach you choose but the consistency in applying it, which is what babies thrive on.
So it's a good time to be a hedgehog and not a fox, just find one book which fits your world view and then follow that guide religiously. This is also much less stressful than juggling all of the conflicting advice you're going to receive.
Have a blast and best of luck on your journey!
The index will pay you beta, which is the market return - no more, no less. If you build a portfolio where you combine an investment in the index fund with other, different investments, you can build many different portfolios which all have different risk profiles. This is useful, as people have different needs.
For example, if you're happy for your money to be locked away for a long period of time (say you're running a university endowment fund), you may be better suited to these longer term, illiquid type of in investments.
Secondly, it sounds like you've lived yr teens a little differently to others, but there's no right way of doing it, and you'll be richer for some of the choices you've made and poorer for others, that's part of life. Live without regrets. It's a cliche but also a truism that you can't change the past.
Finally, the great thing you've done is yr reflecting upon the life you're leading. The trick now is to actually act upon the changes you want to make and commit to them. If you can work this loop really effectively (do, reflect, improve) while also just being able to relax a bit and find pleasure in where you are right now then I think this is one of the keys to a happy, fulfilled life.
Good luck!
It's interesting that he starts out by arguing that he's engaging with popular culture through television and he is not an elitist, but then ends the piece disengaged from the internet and arguing for the intellectual elites as cultural gatekeepers, and against the internet as it bypasses that process.