447 karma · joined May 29, 2008
The algebraic conception, with its wild automorphisms, exhibits a kind of multiplicative chaos — small changes in perspective (which automorphism you apply) cascade into radically different views of the structure. Transcendental numbers are all automorphic with each other; the structure cannot distinguish e from π. Meanwhile, the analytic/smooth conception, by fixing the topology, tames this chaos into something with only two symmetries. The topology acts as a damping mechanism, converting multiplicative sensitivity into additive stability.
I'll just add to that that if transformers are implementing a renormalization group flow, than the models' failure on the automorphism question is predictable: systems trained on compressed representations of mathematical knowledge will default to the conception with the lowest "synchronization" cost — the one most commonly used in practice.
https://www.symmetrybroken.com/transformer-as-renormalizatio...
I wonder if there's a connection to your Taylor truncation order. In RG terms, higher-order polynomial interactions are "irrelevant operators"—they get suppressed as you flow toward the fixed point. If trained attention heads are sitting near this fixed point, that might explain why modest truncation orders work: the network has already learned to concentrate its computation in the lower-order terms. A testable prediction: layers with α closer to 2 (measurable via weightwatcher https://github.com/CalculatedContent/WeightWatcher) might need fewer Taylor terms for accurate approximation than layers with α far from 2. If true, you could potentially use the spectral statistics to adaptively choose truncation order per-head.
Let's assume that the brain implements a renormalization group flow in a way similar to transformers, as described here: https://arxiv.org/abs/2507.17912
In biological neural networks, synaptic weight distributions presumably have some effective α. Learning perturbs α away from optimal — overfitting would correspond to α drifting toward values that capture training noise rather than generalizable structure. The question becomes: what mechanism pushes α back toward 2?
Astrocytes are positioned to do exactly this. They don't modify individual synaptic weights (that's Hebbian learning), but they modulate the gain of synaptic transmission across vast swaths of tissue simultaneously. This is a different kind of operation — more like rescaling the entire weight matrix or reshaping its spectral density than updating individual entries.
Consider what happens during sleep. The synaptic homeostasis hypothesis (Tononi & Cirelli) proposes that sleep globally downscales synaptic strengths, renormalizing the system after a day of potentiation. This is literally a renormalization operation — a rescaling that preserves relative structure while adjusting the overall scale. If astrocytes mediate this process (and the Quanta article suggests they track sleep debt via calcium accumulation), they might be implementing something like the ERG step that SETOL identifies as optimal.
The scale-invariance condition is key. SETOL states that the ideal layer is governed by a Scale-Invariant transformation equivalent to a single step of an Exact Renormalization Group transformation. Astrocytes, by modulating transmission globally rather than synapse-by-synapse, naturally implement transformations that preserve relative structure while adjusting scale — exactly the kind of operation that could maintain or restore scale invariance.
The zebrafish "giving up" experiment becomes interesting through this lens. When the fish swims futilely, neural circuits are active in a regime that's metabolically expensive but informationally unproductive — the current model (swimming should work) is wrong, and the system is far from any useful attractor. The accumulated evidence in astrocytes might track not just "futility" in behavioral terms but deviation from optimal α — a signature that the current operating regime violates the ERG condition.
The state switch, then, isn't just behavioral resignation. It's the system recognizing that its current configuration is far from scale-invariant equilibrium and forcing a transition to a different basin where better spectral properties can be achieved. The adenosine release that triggers the switch would be the biological mechanism implementing what mathematically amounts to a large step in parameter space toward a more ERG-compliant regime.
The deepest connection might be to criticality itself. Systems at critical points exhibit scale invariance — correlations extend across all scales, and the system looks statistically similar under coarse-graining. The α = 2 condition in SETOL might be a signature of criticality in the weight space, just as power-law correlations are signatures of criticality in physical systems.
If brains operate near criticality, maintaining that critical state requires feedback — something must detect drift toward subcritical or supercritical regimes and push back. Astrocytes, with their slow timescales and broad spatial integration, are ideally positioned to monitor deviations from criticality and implement corrective homeostasis.
The Kadanoff-Wilson RG originated in the theory of phase transitions. SETOL adapts it to neural networks. The biological realization might be: neurons implement fast inference (the "flow" of representations through layers), while astrocytes implement slow RG steps that maintain the spectral conditions (α ≈ 2) under which that inference remains optimal.
Testable Predictions This framework suggests several predictions: * Synaptic weight distributions should drift away from α = 2 during extended waking and return toward it during sleep. * Astrocyte-deficient or astrocyte-disrupted animals should show abnormal weight spectral properties — perhaps more variance in α across brain regions, or systematic drift away from optimal values. * The giving-up threshold might correlate with cumulative deviation of local circuit α from 2 — not just behavioral futility, but a kind of "spectral stress" that astrocytes track. * Interventions that artificially maintain α ≈ 2 (if such were possible) might reduce the need for sleep or astrocytic modulation.
Any experts here? Astrocytes as implementers of biological RG steps that maintain the spectral conditions for optimal inference?
Me too! But I was wondering what might be possible more in the next decade. :-D
If we think about stability from first principles, then when you're talking about aggregating up billions and trillions of transactions, each of which occurs with some characteristic frequency, then the primary factors are coupling strength (how correlated or uncorrelated are those frequencies?) and bandwidth (how wide is the spread of their characteristic frequencies).
It's very popular right now to look down on the central banks for the inflationary influence of monetary policy. Less well appreciated is their role in damping volatility through decoupling (by calming herding behavior -- e.g., stopping bank runs) and increasing bandwidth (by offering the longest terms of credit and the ultimate put).
I believe you have to look all the way back to how market panics were resolved by J.P. Morgan (the person) to get a sense of how volatility in a internet-native stable coin might be addressed. I love crypto, but I don't see that crypto has fundamentally changed human nature, which means that ultimately some crypto holder will have to serve as the liquidity provider of last resort. This, in turn, raises all the same questions about centralization vs. decentralization and democratic values that we have with respect to the central banks in terms of how they handle monetary policy.
But maybe there's a way I haven't seen yet.
Isn't Bitcoin (or Ethereum, or Solana, or whatever your L1 favorite is) itself the "internet-native stable coin" we want, except for their lack of stability? In other words, if we didn't have a need to exchange into local currency, what remaining need would there be except for lower price volatility?
Suppose further that corporate managers were prohibited from making tax-advantaged donations to non-profits except upon a fixed percentage of employees approving the donation. Would that help?
Aside from employees, there are also the communities within which corporations operate. Suppose we required that a certain percentage of any taxable profits be earmarked for donation to non-profits within the communities within which the corporations operate. Would that help? But maybe we would we have to pair this with the proposal above for this to work?
One way to address the relative inefficiency of a larger organization is to consume more energy and not worry about the waste on entropy. This works so long as the large organization is growing — i.e., so long as it is able to extract more energy from its environment than it is wasting (in a relative sense) on its internal processes.
The strategies for minimizing entropy within an organization — large or small — seem to boil down to two, which are intertwined: 1) what @pg called "Founder Mode" and 2) alignment around mission and vision. In both cases, the effect is to drive the organization towards a "critical state" in which small details of information picked up at the edges can be shared relatively quickly across the entire organization, allowing every part of the organization to react in alignment to that new information. In the case of 1), this is facilitated by a dictator (i.e., the founder) who everybody willingly submits decisions to when they themselves are unsure of how the founder would decide. In the case of 2), this is facilitated by a shared understanding of what the "right" decision is across the organization in view of the mission and vision, which are clear and crisp enough to answer most questions, even about relatively obscure issues or questions that arise.
The ability to operate at scale seems more or less to be derived from one or both of these. Coase's theory of the firm in The Nature of the Firm can be understood in these terms — that is, 1) and 2) are the mechanism whereby internal management outperforms spot markets in coordinating production.
In fact, I would say this is a key source of the tension between large and small that Terry has identified. Yes, large organizations are more efficient at most of what we need as humans. But our ape brains still benefit from being close enough to smell the people we're working with. Until we evolve biologically, it's going to be a problem that it's so much easier to work remotely than it is to work together in person. And the world is only making it harder to do the latter right now.
I think North, Wallis, and Weingast's Violence and Social Orders is a little more directly on point to his posts, but I believe I share at least part of your point of view here — and would love to see a precis of the magnum opus!
Personally, I see the economic efficiency argument for having a relatively small number (say 2 or 3) large organizations that maintain key "distribution platforms". We don't need twenty different social media websites. Antitrust/competition law can play an important role in ensuring that everybody gets access to these platforms on the same basic terms — no favoritism. But I don't know that we need to prevent acquisitions of new apps that can then be bundled into services offered by one or more of those platforms -- at least not to achieve the balance between "small" and "large" that Terry seems to have in mind here.
Rather, I think what we need to aggressively protect is the incentive that small groups have to form and fund startups. In general, I think we're doing fine on the economic side right now. An exception might be the recent acquisitions of founders independent of their coworkers — this presents a profound threat to the startup ecosystem. But by and large the system seems to be working fine.
But is that true of political "startups" — i.e., new interest groups that form to pursue specific policy agendas, which might then be able to syndicate eventually even into new political parties? Of that I'm less certain. It sure seems like the last year or two have been trending toward less political freedom.
According to North, Wallis, and Weingast, the "double balance" of open political and economic competition — between incumbent large organizations and what we might call "startups" — is what allowed a handful of countries to transcend the "natural state" or "limited access order" in which an oligarchy of elites control the economy and government (including access to violence).
It seems like his posts represent an independent reproduction of a key piece of their theory.
I wasn't familiar with the term "equivariance" but I "woke up" to this sort of approach to understanding deep neural networks when I read this paper, which shows how restricted boltzman machines have an exact mapping to the renormalization group approach used to study phase transitions in condensed matter and high energy physics:
https://arxiv.org/abs/1410.3831
At high enough energy, everything is symmetric. As energy begins to drain from the system, eventually every symmetry is broken. All fine structure emerges from the breaking of some symmetries.
I'd love to get more in the weeds on this work. I'm in my own local equilibrium of sorts doing much more mundane stuff.
It's only an analogy, but it does suggest at least that the interesting details of the dynamics aren't embedded in the topology of the system. It's more complicated than that.
Significant bit from the conclusion: "The theory is testable and falsifiable at the level of computational modeling, but also in terms of neural implementation."
https://modelcontextprotocol.io/quickstart/server
I don't think it's terribly difficult to drill down into their GitHub to see what's happening under the hood if you need that level of detail.
It's hard for me not to think of the electric guitar in this context. I love the purity of an acoustic played in a plangent tone like Django. But the incredibly rich spectrum introduced when an amplifier saturates and tons of additional harmonics and subharmonics that aren't supposed to be there gets introduced -- that's it own kind of excitement. Like falling into an abyss aurally.