versteegen

> They pragmatically changed their views of safety just recently, so those values for which they would burn at the stake are very fluid.

Yes it was a pragmatic change, no it was not a change in their values. The commentary here on HN about Anthropic's RSP change was completely off the mark. They "think these changes are the right thing for reducing AI risk, both from Anthropic and from other companies if they make similar changes", as stated in this detailed discussion by Holden Karnofsky, who takes "significant responsibility for this change":

https://www.lesswrong.com/posts/HzKuzrKfaDJvQqmjh/responsibl...

> I strongly think today’s environment does not fit the “prisoner’s dilemma” model. In today’s environment, I think there are companies not terribly far behind the frontier that would see any unilateral pause or slowdown as an opportunity rather than a warning.

> What I didn’t expect was that RSPs (at least in Anthropic’s case) would come to be seen as hard unilateral commitments (“escape clauses” notwithstanding) that would be very difficult to iterate on.

Sure, depending on the particular product, having control and direct local access to the data would be desirable or deal breaker. But for this Clickup integration that's not so important to us (we can duplicate where necessary), while still using Clickup lets us use all the other features we can get via the web app.

(The emacs mode includes an MCP server)

I'd agree that this effect is probably mainly due to architectural parameters such as the number and dimensions of heads, and hidden dimension. But not so much the model size (number of parameters) or less training.

Saw something about Sonnet 4.6 having had a greatly increased amount of RL training over 4.5.

Agreed, and here's a real example from a tiny startup: Clickup's web app is too damn slow and bloated with features and UI, so we created emacs modes to access and edit Clickup workspaces (lists, kanban boards, docs, etc) via the API. Just some limited parts we care about. I was initially skeptical that it would work well or at all, but wow, it really has significantly improved the usefulness of Clickup by removing barriers.

> The only valid ARC AGI results are from tests done by the ARC AGI non-profit using an unreleased private set. I believe lab-conducted ARC AGI tests must be on public sets and taken on a 'scout's honor' basis that the lab self-administered the test correctly

Not very accurate. For each of ARC-AGI-1 and ARC-AGI-2 there is training set and three eval sets: public, semi-private, and private. The ARC foundation runs frontier LLMs on the semi-private set, and the labs give them pre-release API access so they can report release-day evals. They mostly don't allow anyone else to access the semi-private set (except for live Kaggle leaderboards which use it), so you see independent researchers report on the public eval set instead, often very dubious. The private is for Kaggle competitions only, no frontier LLMs evals are possible.

(ARC-AGI-1 results are now largely useless because most of its eval tasks became the ARC-2 training set. However some labs have said they don't train LLMs on the training sets anyway.)

Paid/API LLM inference is profitable, though. For example, DeepSeek R1 had "a cost profit margin of 545%" [1] (ignoring free users and using a placeholder $2/hour figure H800 GPU, which seems ballpark of real to me due to Chinese electricity subsidies). Dario has said each Anthropic model is profitable over its lifetime. (And looking at ccusage stats and thinking Anthropic is losing thousands per Claude Code user is nonsense, API prices aren't their real costs. That's why opencode gives free access to GLM 4.7 and other models: it was far cheaper than they expected due to the excellent cache hit rates.) If anyone ran out of money they would stop spending on experiments/research and training runs and be profitable... until their models were obsolete. But it's impossible for everyone to go bankrupt.

[1] https://github.com/deepseek-ai/open-infra-index/blob/main/20...

It's excellent that you're working on loneliness! Somehow. What is it your startup actually does?

Yes, unfortunate that people keep perpetuating that misquote. What he actually said was "we are not far from the world—I think we’ll be there in three to six months—where AI is writing 90 percent of the code."

https://www.cfr.org/event/ceo-speaker-series-dario-amodei-an...

That's pretty good, I'm jealous! The last time I reinstalled my OS (Slackware) from scratch was 2009, but I run into serious problems every couple of years when upgrading it to 'Slackware64-current' pre-release, because Slackware's package manager doesn't track dependencies and you can just install stuff in the wrong order: I usually don't upgrade the whole OS at once... just have to fix any .so link errors (I've got a script to pull old libraries from btrfs snapshots). I've even ended up without a working libc more than once! When you can't run any program it sure is useful that you can upgrade everything aside from the kernel without rebooting!

Then what do you say to 6.14" × 2.61" × 0.11 mm = 102 cm³

Yes, it's a great deal especially because you get access to such a wide range of models, including some free ones, and they only rate limit for a couple minutes at a time, not 5 hours. And if you go over the monthly limit you can just buy more at $0.04 a request instead of needing to switch to a higher plan. The big downside is the 128k context windows.

Lately Copilot have been getting access to new frontier models the same day they release elsewhere. That wasn't the case months ago (GPT 5.1). But annoyingly you have to explicitly enable each new model.

I have no idea at all whether the GCP "Service Specific Terms" [1] apply to Gemini CLI, but they do apply to Gemini used via Github Copilot [2] (the $10/mo plan is good value for money and definitely doesn't use your data for training), and states:

  Service Terms
  17. Training Restriction. Google will not use Customer Data to train or fine-tune any AI/ML models without Customer's prior permission or instruction.

[1] https://cloud.google.com/terms/service-terms

[2] https://docs.github.com/en/copilot/reference/ai-models/model...

That's an interesting observation. I'd suggest modelling the LLM's behaviour in that situation as selecting between different simple strategies, each of which has its own transition function. Some of the strategies will be far more common than others. Some of them may be very simple and obey the detailed balance condition (meaning they are reversible Markov chains), but others, and the overall transition function does not.

The definition of the detailed balance condition is very strict and it's obvious that it won't be met in general by most probabilistic programs (sets of rules with probabilistic output) even if you consider only those where all possible outputs have non-zero probability (as required by detailed balance).

And the LLM+agent is only a Markov chain because of the limited state space of the agent. While an LLM is adding to its context window without reaching the window size limit, it is not a Markov chain, as I explained here: https://news.ycombinator.com/item?id=45124761

And, agreed that better optimisation would be incredible. (I would describe it as a search problem.) I'm not sure how feasible it is improve without changing the architecture, e.g. to a diffusion language model. But LLMs already predict many tokens ahead at once which is why beam search is surprisingly unnecesarr. That's how they're able to write coherent sentences (and rhymes), they've already largely determined at the beginning what they're going to write. (See Anthropic mech interp work.) So maybe if we could tap into that we search over vaguely-formed next blocks of text rather than next words.

Since it took me some minutes to find the description of the task, here it is:

We conducted experiments on three different models, including GPT-5 Nano, Claude-4, and Gemini-2.5-flash. Each model was prompted to gener- ate a new word based on a given prompt word such that the sum of the letter indices of the new word equals 100. For example, given the prompt “WIZ- ARDS(23+9+26+1+18+4+19=100)”, the model needs to generate a new word whose letter indices also sum to 100, such as “BUZZY(2+21+26+26+25=100)”

One interesting and ironic part of the article is that one of the mentioned optics research groups has been submitting a lot of patents on EUV sources. Are we meant to be mad about it?

So how do the models compare in your experience?

Tip: the crashing is caused by certain extensions such as OneTab and All Tabs Helper which for some reason seem to cause all the tabs to load, just when restoring a session. Temporarily disable these extensions before restoring, then you can reenable.

> - It should even demonstrate consciousness.

I disagreed with most of your assertions even before I hit the last point. This is just about the most extreme thing you could ask for. I think very few AI researchers would agree with this definition of AGI.

> A junior developer has no such skills. Their only approach will be to run the code, test whether it fulfills the requirements, and, if they're thorough, try to understand and test it to the best of their abilities.

This is also a supremely bad take... well, really it's mainly the way you worded it that's bad. Juniors have skills, natural aptitudes, as much intelligence on average as other programmers, and often even some experience but what they lack is work history. They sure as hell are capable of understanding code rather than just running it. Yes, of course experience is immensely useful, most especially at understanding how to achieve a maintainable and reliable codebase in the longterm, which is obviously of special importance, but long experience is not a hard requirement. You can reason about trade offs, learn from advice, learn quickly, etc.

A completely flipped perspective:

> "Why the f*ck are you asking, you should know this"

Because you mentioned NZ: my father, a toolmaker, said there was a huge difference between Europe and NZ. In Germany/Netherlands, he'd be working under a more senior toolmaker. When he took a job in NZ and asked the boss something, as would have been the proper thing to do in Europe, he got a response just like that: because he was the expert, and his NZ boss was just a manager.

> citations from a wide range of dates

Ouch, really? That's basically just work that's not in the current hot topics. Not really a datapoint in favour of 'crank', rather a point against 'active academic/student'. I think it's admirable to look for value in older work.

It's silly of them to say you need a "modern terminal emulator", it's wrong and drives people away. I'm using xfce4-terminal.

Gemini 3 via any provider except Copilot should work in Opencode.

Well Opencode also completely replaced its TUI a few weeks ago too.

BTW Gemini 3 via Copilot doesn't currently work in Opencode: https://github.com/sst/opencode/issues/4468

Quoting a month old post: https://www.lesswrong.com/posts/prSnGGAgfWtZexYLp/is-90-of-c...

  My current understanding (based on this text and other sources) is:
  - There exist some teams at Anthropic where around 90% of lines of code that get merged are written by AI, but this is a minority of teams.
  - The average over all of Anthropic for lines of merged code written by AI is much less than 90%, more like 50%.

> I've never gotten a service overloaded response from Google like I did from Anthropic

They're Google, they out-scale everyone. They run more than 1.3 quadrillion tokens per month through LLMs!

I'm really confused by "about $15k of x86 gear ... The minute AMD K10 and Intel Nehalem dropped, high routing performance was mostly a software problem". What kind of $15k machine would you have needed? That's a heck of a lot more than even the most expensive K10 2008 CPU (which according to Wikipedia seems to be Opteron 8384 (quad core, 2.7GHz, 1.0GHz HT, $2149 November 2008), supports up to 8 CPUs per machine, I guess that's what you mean.)

Imagine I capitalised a whole selection of specific words in this sentence for emphasis, how annoying that would be to read. I'll spare you. That is what the guideline is about, not one single instance.

I thought the repetition was intentional.

Well, I also work on AI, and I completely agree with you. But I've reached the point of thinking it's hopeless to argue with people about this: It seems that as LLMs become ever better people aren't going to change their opinions, as I had expected. If you don't have good awareness of how human cognition actually works, then it's not evidently contradictory to think that even a superintelligent LLM trained on all human knowledge is just pattern matching and that humans are not. Creativity, understanding, originality, intent, etc, can all be placed into a largely self-consistent framework of human specialness.

No I didn't mean to draw attention to HFTs specifically, I understand they aren't big. I said 'second' instead of 'subsecond' because humans are also capable of reacting within seconds. But you are right that all those day traders losing money even things out and I was concentrating on the wrong thing. Investing and trading on longer time scales is certainly profitable.

Obviously the financial sector provide a lot of value, but they also extract a LOT of value and probably even worse employ a LOT of the smartest people (I recall the rebalancing of Iceland's economy after its banks failed), and couldn't we get nearly the same benefits with far less of the global economy being dedicated to financial services and trading (which neither I nor it seems the OECD categorise under "services")?

For example according to the OECD [1] 25% of Luxembourg's GDP (excluding interest and trading profits [2]) and 10% of employment is due to financial services! For comparison, for the UK and USA it's 8.8% and 8.3% of GDP.

In particular it's hard to me to see how market trading activity that provides a price for equities to the second, instead of say holding auctions every hour (which would probably greatly reduce profits for day traders and HFT), helps any drug development or aircraft leasing company to raise money. Financing deals don't happen on the market, and if market price is involved, typically something like the last month's average daily closing price is used.

We might call middlemen parasitic if they extract more value than they provide, but as you say, without finance the global economy wouldn't function. Let's instead consider the marginal utility of more of the economy being dedicated to finance. I'm convinced it's negative.

[1] https://www.oecd.org/en/publications/2025/04/oecd-economic-s...

[2] Quote from [1]:

> In the national accounts, financial services output is measured as the sum of financial intermediation services indirectly measured (FISIM) and fees, for instance on account keeping, credit cards, brokerage, financial advice and asset management. ... Trading profits and other interest income, for instance on bonds and derivative products, are excluded from the national account measure of financial services output.