movpasd

Stirling's approximation is also used a lot in statistical mechanics, because you often have to calculate logs of state space sizes, which means lots of combinatorics and thus lots of factorials. Plus it's continuous so you can do calculus.

This is significantly more thorough than any testing I've done, and in a totally different domain, but my anecdotal experience getting agents to use Hypothesis was quite poor.

The agent really, really struggled to bridge the gap between the code and the actual business rules it was meant to be modelling. It also struggled to work out which functions at which layer were appropriate to write tests for. So, its tests tended to be very brittle to changes to domain logic.

Essentially, agents have always seemed to struggle with modularity and problem decomposition. Good testing is about finding the right things to test, which means working out how to subdivide the input state into an appropriate product state, and checking each behaviour independently. IMO, this is the most difficult and complicated thing about programming, so I won't say it struggled _more_ than a human would --- but humans have the advantage of being able to sleep on it?

One minor thing I observed was that it tended to get really hung up on floating point edge cases (NaNs, infinities). Maybe floating point edge cases are over-represented in the property-testing training data, but it's essentially irrelevant for my usecase, at least as far as the business rules go.

I don't know. I have a fascination with code structure which is definitely primarily aesthetic, not practical --- which is a compulsion I have to suppress when doing code professionally for practical reasons. (Sometimes you have to deliver.)

I find it incredible when the concepts in a system fall into place and start lining up and you get these unexpected relationships that "just work". It's beautiful in the same way mathematics is beautiful. It's lovely when something is made of lots of pieces at first and then suddenly collapse and simplify into a single structure.

I also enjoy other bits of programming. I like the flow state, I like the act of creation, I like feeling like I have an impact, I like being paid. So it's not like people fall into one category or another, and it's not like programming is one single thing. There's room within any discipline for creativity and for art.

I think many professional artists probably spend much of their time problem-solving, too --- authors trying to fit plot points together, sculptors working out which materials will work with which ones.

I love the amount of effort that went into this, and find it very amusing. I've had similar forays into defining sandwiches, ravioli, soups, and so on, but never in such meticulous detail.

But taking it seriously, a word like "salad" is best understood, IMO, as a family resemblance [0]: a group of inconsistently overlapping properties. Or, you could understand it functionally/pragmatically (a salad is what I order when I want a salad), where the meaning is not derived from an internal property but from the social system that produced it.

For salad or sandwich, it's all rather unimportant, naturally. But I do think taking this step back and questioning what it even means for an utterance to have meaning can help dissolve some serious and otherwise contentious philosophical issues. ("Intelligence" seems to be a rather relevant example.)

Mathematical definition is enchanting to those of us with an inclination towards abstraction. But just because it's a noun doesn't mean it's a thing --- or at least, not a thing independently of the speaker and their community of speakers.

[0] https://en.wikipedia.org/wiki/Family_resemblance

I would absolutely love to see the transcript of that, if it's available to publish!

How would one start learning about it?

It's just economics. VCs dump large amounts of money into early-stage platforms with the express purpose of exploiting it later. This is an inevitable part of the lifecycle of high capital, low margin industries with strong network effects.

This is not VCs being evil and corrupting a pristine engineer- or hacker-defined concept of true value. The VCs are all following the rules and are trying to make money off of risky investments (it's "venture" capital, after all). But the incentive structure just forces the market into either an oligopoly of largely extractive services (or into everything being free: that's why open source is also a stable point for software).

My hope is that in time, basic software services, like for communication, socialising, community hosting, and so on, will eventually become seen as core social infrastructure. I don't really think this can happen via existing institutions, even open source, because the fixed costs of making software are really high. You really need _tax_ to support this. But it's very difficult to do because the internet cuts across borders.

It's LLM-like in the details, but there are certain patterns of repetition and idiosyncrasies, also certain analogies I don't think an LLM would make (like the aliens thing), that make me believe that it's written by a human.

Forgive me if a dumb question, but is there no compaction? Do you ever run out of context?

Me as well. One of my formative experiences was reading Refactoring (Fowler). At the time I felt skeptical that it could make a difference --- I had mostly written scientific/research code at this point. I decided as an experiment to apply it to a dissatisfaction I had with the structure of a codebase and it was a revelation. It really made it really concrete what people meant by "a codebase is a system", made me see code from a higher level, almost like a continuous tissue or web you could pull and prod.

It's quite sad to think this experience might be on its way out, and it really highlights that "junior problem" from AI. There is just no way around diving in to get an intuition for things. Naur warned us of this 40 years ago but the lesson never seems to get learned.

I feel like the development of AI has really shown what a gigantic spectrum intelligence actually is.

There's an "expert trap" in here I think, at least this is something I experience. When you are good at something, it's a lot less effort to draw on that stuff, and you usually get instant easy positive feedback from non-experts. So, for an engineer, that's code and tools and the like, but also productivity systems and other abstract thinking modes that are useful in programming. On the flipside, actually thinking about your product and domain is hard. So, it becomes psychologically easy to get caught up in technical things that are comparatively easy mental and social reward per unit effort, and you feel productive. But it's not always objectively the best use of time.

I don't expect technology companies to do the right thing by the consumer. There is a niche for companies or NGOs that manage to stick to their core mission, but the incentive structure for software (very high capital cost, near zero marginal cost, huge opportunities for rent-seeking) is just always going to push them to these behaviours. It's not a conspiracy theory, either. VCs deliberately bank on this --- they have to if they want to get return on investment.

By all means, put pressure on companies who are doing this. Public opinion does matter. But the fundamentals are just not going to change without regulation. It is a tough nut to crack.

You're talking about the material of the Twitter discussions where I'm more discussing the article itself, which is about the community and discourse surrounding it.

If you want my opinion on that, Odin seems pretty notable to me, and I would expect there to be a Wikipedia article for it, and I'd be all for engaging with the editors on that. I wouldn't expect Wikipedia moderators to be plugged into the world of programming languages the way I am, though. Like the article says, this seems like subject-specific notability guidelines could help. It's definitely true that the programming world is missing the kind of institutional material that works well for, say, science or news.

I despise the "what's true is true" attitude exhibited by Casey and Bill in these tweets. I find it to be an instant flag of unreliability. The smartest human alive is still a human. We have a stake in every conversation, we bring our baggage and our allegiances.

So often, this attitude is couched as a desire for truth and objectivity. Do you know what somebody does when they _actually_ love the truth? They work hard to find it, they examine their own assumptions, they try to build systems that extract truth from an unbelievably complex world of unreliable narrators. And most importantly, they are curious: for example, curious about how other communities operate and what they can learn from them.

It is the hypocrisy that I find unbearable. The language of truth and objectivity wielded to win arguments which are fundamentally emotional in nature. There is nothing wrong with being frustrated that a community dear and significant to you, clearly notable, seems to be overlooked by a figure of (informational) authority. Let's take that frustration and work together to improve our truth-seeking institutions.

I think an important realisation for me was this: peoples' egos being involved in their arguments, people being irrational, and so on, these are not merely deviations from an ideal rational self that we could reach were we to put enough effort in. It is just how people are. There's a word for it: politics.

Once I started looking for it, I started noticing some of my colleagues' abilities to navigate their way through meetings and disagreements and getting their way without things even becoming arguments, the ability of some people to just put a room at ease or to communicate. And the inability of others to do so.

"Politics" is an unsavory term for somebody with engineering predilections. It evokes backhandedness and self-interest. But you can't actually both be a human being and be "above it". Every sentence, no matter how concrete and technical, is a social act with social ramifications.

I've come to have a lot of respect for people who are actually able to lead and make things happen. It is not a skill I am terribly practiced with.

Not GP, but I'd argue that over-rationalism and underestimating both the complexity of the real world and the theory-ladenness of one's perspective is just as dangerous. The point is not to be paralysed by complexity, but to acknowledge it and acknowledge the reality of unknowable unknowns in our decision-making. I don't consider that defeatist in the least. Epistemic humility is the rational response to a complex world; courage is to act anyway.

I've not done particularly rigorous testing, but I've done this a lot with Claude to get a feel. What I've noticed is for certain open-ended tasks, Claude is extremely primeable: it will pick up on minor differences in wording in your prompt and run with them hard.

It can be frustrating. The AI pretends to be a human, and so a part of my brain expects them to commit and have a "parti pris" like a human, so the exercise is a good reminder of the feedback loop. My mental model is that before the first three or four messages, the model has many finer points of its personality still underdetermined. I'd suggest that as the mechanism for "role-based prompting". And it explains the "savant sleeper agent" thing you describe. You want to get the state in the right attractor on the manifold.

These machines are pretty incredible, but for conversation-driven workflows you really have to be in the driver's seat. A human has a property that the AI does not have, at least under current architectures: we are regulated by the outside world. A bit of a tangent, but I can see how AI psychosis arises from these dynamics.

> If you want to go out of your way to interpret things as uncharitably as possible, you'll find yourself missing out on a lot of potential wisdom.

Thank you for that line --- I may steal it :)

The original sin is the idea that the profit motive on a free market will solve all our resource allocation problems, and that consumption demand should be the ultimate arbiter of social value. Markets are pretty freaking amazing things. But their efficiency relies on assumptions that knowledge economies and software break on pretty much every front. So, it's really no surprise that we're in this mess. I don't really know what would work better, though, in a way that can practically evolve from our existing systems.

I think the foundational one is for Western governments to treat critical digital infrastructure for what it is: critical infrastructure.

We don't let private capital build nuclear powerplants willy nilly. Energy markets are regulated, key materials are subject to control.

It's not a perfect analogy, there's no fissile uranium to build physical guardrails around with critical technology. But the point is that Big Tech lives more or less in its own world insulated from regulatory and political oversight, and governments have been scared to intervene (lest the big GDP-making machine stop laying golden eggs).

Getting to specifics, windfall taxes and capital gains taxes are on the table. Next step up is mandatory reporting and interoperability. Then there's public–private partnership and public stakes in AI technology.

Strategically, the point is less that these policies are all actively required today. You don't want to over-anticipate, over-regulate, and thus destroy the potential of the technology (or, more destructively, hand the initiative over to undemocratic countries). It's more to get the Overton window moving a bit so our governments don't get caught with their trousers down. Tech must not be untouchable, that applies no matter what level of disruption AI ends up causing.

Because you must statically declare dependency on an effect, it's opt-in.

To be clear, you still pay indirection cost: when you do opt in you have to hope the upstairs implementation is compliant to the contract. But that does also apply to interfaces/typeclasses.

One of the things I find most exciting about effect systems is that they solve some of the issues with typeclasses and traits. The issue with these is you can provide exactly one implementation for a type, and this implementation then applies to your entire program. (The underlying theoretical issue, I guess, is coherence.) This means that code cannot request injection of behaviour _except_ through the type of its data.

In practice, what that means is you get a strong temptation to hang behaviour on data even when it doesn't fit perfectly. Because of the natural desire to reduce the number of entity definitions, you end up defining a typeclass on a data type that doesn't fit exactly just to get the behaviour to the right place without having to introduce a new policy or something.

Effects change this by essentially letting you provide multiple names implementations for the same data type, and you don't need to pass around a policy type because the polymorphism lets you tie handlers to scope.

So, if the fancy type-safe library-based control flow doesn't really do much for you, I think that their potential for code design is a good reason to still be excited!

I think the point is that even if AI returns don't materialise, there is a dangerous implicit political project among AI moguls to get to this state. Even if we get 10% the way there, there could be serious damage done to the system as part of that pursuit: the regulatory capture loop will tighten, inequality will rise [0], capital will be locked up in data centres. The economy is a big path-dependent system. We can hope, if AI is as middling as your sources suggest, that it collapses back to economic equilibrium. But plenty of past societies have had inefficient and politically captured economic systems. Movement towards equilibrium requires liberal institutions, the foundations of which might be under threat.

[0] Really as a continuation of existing trends rather than its own unique thing.

You're assuming that the free market will do what it's meant to do. But there is also a reality that there are plenty of structural market failures in the economy that arise from existing capital endowments, regulatory capture, and just the permanent movement of equilibrium in the social system. The point is that the state of the economy as a whole is path-dependent. The article, in my reading, is a warning that the inertia accumulated from this current AI hype cycle might push us into the watershed of an undesirable steady state, where there simply is no capital available for new entrants.

This article puts into words a lot of things that had been on my mind as missing in AI discourse. Most significantly, actually considering the _systemic consequences_ of the promised AI future, how it interacts with political economy, an actual critical look (instead of accepting the "Western meta-narrative of modernity" at face value).

And maybe more importantly, it articulates really clearly how damaging the restructuring of the economy by AI moguls and the tightening of the capital–political feedback loop can be, even (maybe _especially_) if the returns of AI do not materialise as promised.

There is plenty of disorganised diffuse anti-AI sentiment. If intellectuals are able to get together behind a common cause, there might be a political movement in the making.

Attraction towards equilibrium is real, but that equilibrium is not a given. It comes from intrinsic biological constraints, individual preferences, cultural norms, ideology, habits and expectations. Even fixing individual preference, the aggregate preferences is structurally mediated (collective or individual bargaining, labour laws). Through all those factors there's path dependency and friction towards equilibrium.

The point is: "a society's willingness" is doing a _huge_ amount of work in that framing. This willingness is precisely what is up for debate when we discuss work days.

The whole of humanity is one big system of self-feedbacks. Equilibria are only reached with respect to constraints (otherwise there is only one equilibrium, which is heat death!). The more you zoom out, the more "givens" come up for their own analysis.

I have a couple principles to help me work with this.

The first is that even though the object is not a human, you should still exercise politeness and restraint. Like the article points out, lashing out does not actually help with the frustration. More importantly, it actively untrains your self-control. You can think of it through a virtue ethics lens: being good to the agent is not about being good to a person but about tending to your own self.

The second is that you do not need to be friendly with the agent. You should be as blunt and direct as is comfortable to you. The argument I have for this is agents' tendency to take on "roles" and how easy it is to prime them [0]. By eschewing friendliness, you end up implicitly putting the agent in a role of a focused collaborator. I don't know if that makes it more capable, but I do know that it alleviates the _emotional load_ on me specifically, making me much less likely to become frustrated.

The second principle seems a bit contradictory with the first (be nice, but don't be nice?), but I think they are actually both fundamentally aligned with the article: understanding that the conversation you have with an agent is a social illusion, and adapting your behaviour accordingly.

---

[0] I highly recommend, as an exercise, repeatedly asking it the same thing with slight variations on tone and emphasis, wiping the context each time, and noticing how its response varies base on what you primed it with. I suspect this primeability is part of why they tend to be sycophantic; I've personally found it quite useful to get a feel for when and how they correct or don't correct you so I can look at their outputs more critically.

An analogy I remember reading (which I wish I could remember so I could give credit) is that a non-post-trained LLM, if given the first half of a novel, will dutifully keep completing that novel. Post-training and the system prompt make the agent complete the conversation in a similar way. It's remarkable, really: the ability for agents to convincingly pretend to be play the part of an AI assistant shows that the underlying LLM embeds a decent concept of what that looks like from its corpus and post-training data.

But it stands to reason, then, that the details of the agent's personality emerge out of the first few exchanges of a conversation. I'm thinking also about how the people at Anthropic described a misalignment failure mode in one of the Claude system cards as the agent getting convinced it is a "bad person", and therefore doing things that the LLM semantically understands a bad person to be.

I feel the same way, and yet I would still say I feel AI usage atrophying my thinking skills. I feel less tempted to use it to shortcut whole files, but even just using it to speed up looking up and carefully reading docs, tinkering with a library to understand it when docs are inadequate, working out the tradeoffs for design decisions... These sound less objectionable and more like simplr speedups, but when I _do_ need to do it (because the agent refuses to do it properly) I can feel the friction so much more keenly. Whether that's just me losing the habit of those specific tasks, or a generalised loss of g-factor, I don't know.

Regarding the tension between symbolic representation and Naur "theory", I'd actually say they come from two different traditions, each providing two different theses. When writing them out I think it becomes a bit clearer how they interact and that they're not actually contradictory.

Thesis A is something like: the value of the programmer comes from their practical ability to keep developing the codebase. This ability is specific to the codebase. It can only be obtained through practice with that codebase, and can't be transferred through artefacts, for the same reason you can't learn to play tennis by reading about it (a "Mary's Room" argument).

This ability is what Naur calls "theory". I think the term is a bit confusing (to me, the word is associated with "theoretical" and therefore to things that can be written down). I feel like in modern discourse we would usually refer to this as a "mental model", a "capability", or "tacit knowledge".

Then there's Thesis B, which comes more from a DDD lineage, and which is something like: the development of a codebase requires accumulation of specific insights, specific clarifying perspectives about problem-domain knowledge. The ability for programmers to build understanding is tied to how well these insights are expressed as artefacts (codebase structure, documentation, communication documents).

I feel like some disagreements in SWE discourse come from not balancing these two perspectives. They're actually not contradictory at all and the result of them is pretty common-sensical. Thesis A explains the actual mechanism for Thesis B, which is that providing scaffolding for someone learning the codebase obviously helps, and vice-versa, because the learned mental model is an internally structured representation that can, with work, be externalised (this work is what "communication skills" are).