7,671 karma · joined February 26, 2017
- Anthropic has a cult-like culture. AI Safety is their religion. The AI Constitution is their bible. Dario is their cult leader. Employees are the apostles. They just really really believe their church, and only Dario is qualified to manage the AI safety.
- Asymmetrical risk. If Dario speaks optimistically about AI and he turns out to be wrong, he'd face the rage of many people. If he fans the doomerism of AI and he turns out to be wrong, at most he will be mocked.
- Regulatory capture. After all, pretty much all the AI big shots in Biden government went to Anthropic. They produced the Biden's regulation, and they made it clear that they wanted to pick a few winners to back.
Instead, they act more like highly technical product managers. They help VPs plan and write high-level product requirements sprinkled with technical terms. They draw boxes on whiteboards and create pretty slides. They write polished documents that keep their leadership happy, and they are either in meetings or on their way to the next one. When they have a technical idea, they dispatch a team to test it out.
Naturally, they still feel deeply technical, until the day they have to resolve a production issue, pass a technical interview, or write extensive code. That is when they realize their skills have grown rusty.
I point this out not to criticize, but to highlight a genuine career challenge. As an engineer, I would rather hone my technical skills. Yet, if you want to climb the corporate ladder, you have to take on more organizational work. The only solution I can think of is to become more like a researcher or a professor. Over the years, good professors spend less time writing papers or deriving formulas. However, their insights are so deep that they still produce amazing results by advising PhD students. But that path is much easier said than done.
April: Anthropic announces Claude Mythos. Citing safety concerns regarding the model's capabilities, they restrict access to a small group of testing partners.
May 15: Anthropic executive Chris Olah visits the Vatican to assist the Pope in publishing a 40,000-word manifesto, Magnifica Humanitas. Olah publicly states the model exhibits over 100 distinct emotional traits.
The pre-release: Anthropic briefly calls for a global halt to AI research.
The release: Anthropic launches Claude Fable 5, a consumer-facing version built on the Mythos architecture but with strict safety guardrails.
The post-release throttling: Users discover Fable 5's performance is being silently degraded on specific tasks. Anthropic acknowledges the throttling, citing the need to prevent foreign competitors from extracting synthetic training data. They promise future limitations will be transparent. Performance is notably restricted on biology and chemistry prompts.
The essay: CEO Dario publishes an essay advocating for strict government oversight. He proposes heavy regulation for any AI company exceeding $500 million in AI revenue or $1 billion in R&D spending.
The regulation: Two days later, the US government places export controls on both Mythos and Fable 5, classifying them alongside high-end semiconductor equipment. This immediately blocks Anthropic's own non-US employees from accessing the models.
The fallout: Amodei calls the export controls a "misunderstanding" that disrupts their global internal operations. Anthropic subsequently issues a statement noting that known jailbreaks for Fable 5 are also effective on OpenAI's GPT-5.5, highlighting the lack of equivalent regulatory action against their competitor.
It looks those guys are acting on asymmetrical risks: if they speak optimistically about AI yet they are wrong down the road, they may get angry mobs. If they speak pessimistically about AI and they are wrong, the worst is that they get mocked.
In all seriousness, though, I'm indeed curious about Anthropic's engineering practice, particular how they can achieve such level of autonomy.
> Quotas are handed down to managers for the splits of the workforce to be put in each ‘bucket’, and the internal politics gets heated as managers try to get their reports into higher buckets.
Curious, why can't the management assigns budgets (or resources in general) to individual teams? That is, it is the managers who are responsible for the resources that their teams get, and the budget is tied to the importance of the "team" that each manager owns. In that way, all the performance review will be local to each team. As a manager, I'd be responsible for the output and importance of my team, and I answer to my manager because they will allocate the budget (or resources in general). Recursively, my team members will answer to me and I don't have to justify who gets rewarded by how much to my peers, except that there will be some form of check and balances.
How Meta manages their perf review seems to set up their managers to be ineffective.
I'm not sure how this matters compared to the other platform companies. Kindle has such a small niche market and the Kindle "platform" hardly registers any impact.
This is what Netflix has always been advocating for. Reid Hoffman also wrote the book The Alliance, in which he argues that employees and employers are allies. When they are aligned, they work together. When not, they part ways.
I find that these two views are realistic and we can use them to guide our actions.
[1] Unless you're an engineer in Anthropic, so you just spend you time writing "loop".
It's actually not that hard to explain if we take into account what Dario kept saying: he, or Anthropic thereof, would be the gatekeeper. It is he who tells the government how to use Claude to design drones. It is his model that tells users whether they can ask a question to Claude or not. And it is he who can assess whether a jailbreak is dangerous or not.
Personally, I think that is way more dangerous than being a hypocrite. Dario is basically the Robespierre of the AI era. He believes that only he gets to decide whether our thoughts, or our prompts thereof, are pure. Anything impure gets purged. For his moral utopia to stand, he has to wield the guillotine. Otherwise, with the chaotic diversity of human nature, how else do you manufacture that perfectly uniform, beautiful morality?
I find it amazing how robust the current deep learning models are. A simple linear combination of every weight did not degrade the performance of the model, but enhanced it.
Don't get me wrong. I hate war. And never-ending wars like the Iraq War anger me to no end (and for that matter, I think G.W. Bush and his cabinet were truly evil). Of course, the danger is real; a military built for defense can easily become an instrument of tyranny or empire if left unchecked. That is why we must maintain rigorous civilian oversight and strict checks and balances over its power. But that does not mean the military, by default, is always evil, right?
I wonder if any tech company managed to thrive long in history by betting so violently on fear mongering and regulatory capture.
Not to be sarcastic but just to offer an observation: in a sufficiently large or bureaucratic organization, preventing an incident from happening can rarely get you any credit or visibility. Such achievement falls into the bucket of "what you're supposed to do". So, those who navigate company dynamics well would rather let the incident happen and then be loud on the follow-up action items. The trick is not to turn an incident into a diaster, so it's a dedicate act.
People think otherwise with AI partly because Anthropic kept telling us that they didn't have to write code or review code any more for most of their work. Their agent swarms just comb through their github, slack and wikis to figure out what to do next, and another swarm of agents just review, test, merge, deploy, A/B test, and revert the code. Boris alone merged nearly 300 PRs in the past week (or two?). So the top research labs seem have broken the productivity seal.
And then they talk about this recursively self-improving AI that is so powerful, so autonomous that they advocate that every company should be prepared to "pause" the effort. And their Fable/Mythos has this specific restriction as mentioned in their model card[1] that they are going to reject requests to tune and train models because, well you guess it, the models are too powerful to be used by mere mortals.
[1] We’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design). Using Claude to develop competing models already violates our Terms of Service, but enforcing this restriction through our safeguards avoids accelerating the actors most willing to violate these terms. Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).
If we don't consider the potential loss of our jobs, on the other hand, isn't it great that we don't have to repeatedly do what we already know how to do? I mean, how many times can we feel the thrill by writing the same CRUD applications? How many times do we have to design the same idempotent APIs? It's also a relief that we could spend way less time figuring out mitigations or root causes when there is a production incident.
This reminds me of the scribes before Gutenberg's moveable-type printing press. They spent their life in scriptoriums copying the manuscripts by hand. They earned three times of the average income of their times. They were highly skilled labor. It required years of training, deep literacy, and a high level of domain expertise. Yet, history showed that even highly specialized expertise can be mechanically reproduced.
That appears to be exactly what LLMs are doing for us: automating the digital equivalent of manual transcription, such as setting up the repetitive boilerplate, sketching out the standard APIs, finding predictable bug fixes.
I'm not sure about others, but I have to face the same existential question today: as software engineers, where does our true value lie? Is it merely in learning, memorizing, and, reproducing patterns that others have already built. More often than not, patterns that an LLM can now piece together better and faster? Or is it in taking everything we’ve learned and applying it to solve entirely new, messy, and uniquely human problems? If our worth is tied to how well we copy the past, we are already obsolete. Our value has to shift from being human repositories of known solutions to being creators who venture into the unknown.
It is, of course, easier said than done. Hence I have likely the same level of stress as other software engineers.
I suspected it was more likely that the intellectual work had already been done in a similar way by a number of other people, and GPT-4 picked up that work.
I realized it long ago: one needs output to make meaning. Input can only be the cherry on a cake in one's life. That, actually, makes FIRE or Fat FIRE not so sustainable unless one has other hobbies.