The Decoherence Trap | Hacker News Reader

The Decoherence Trap

maevyn.codes

6 pointstaybin6 comments

This argument is true as long as coherence can only be derived from the state (or context) of the outside world. As soon as coherence becomes an inherent and measurable property of systems themselves, AI can be theoretically trained inside a bubble to achieve high amounts of coherence. Adapting the systems to the real world is then part of the inference procedure, but no longer the training procedure. Since the article argument is based on a plotting of coherence inside graphs, it basically suggest that coherence may be a system property on its own together with clear, abstract rules to derive it.

The issue is that you are always optimizing for the current state of the systems, not the _next_ state of the systems, within the bubble. You may be able to increase coherence for the current known set of problems, but you cannot know if that coherence will hold for the _next_ problem.

This is essentially the same issue as the halting problem. You don't know what you don't know, and you don't know what happens if you go just one more loop or explore just a little more about the universe, if all of the previous structures hold. Black swans happen all the time, and adaptability is about working within that matrix of data, absorbing it, and reconfiguring around it.

I have to disagree. If there exists an underlying abstract, general pattern of coherence, you can optimize against it without the need of any current or next real world state. The next real world state is then resolved during inference by a system that produces high coherence. But the training can be independent. I am not claiming that there exists such learnable signal - i have no clue either, but wanted to note that the article somewhat suggests this in its plots.

I would argue that there isn't an underlying general pattern of coherence, at least not one that is measurable in a simple way. The simulation in the article simplifies representations of the world into nodes and edges, but reality is often much more nuanced and complex.

I think the section on Futures really points out how difficult problems can be. They were first introduced in the 1970s, they were implemented in the 80s. They didn't become super relevant until the late 80s/early 90s, and even then it took another 20 years of data gathering, experimentation, and distillation across the entire academic domain AND the industry and all of its real world use cases to really nail them down.

This is basically similar to how prices act as a decent signal of value to a point, but break down because they destroy information about what was actually valuable by simplifying it. The same would be true of an abstract measurement of "complexity" when applied universally to all problems, unfortunately.

I agree that a formalization of coherence (if it exists) probably would be more complex than a few constraints on a simple graph representation. However, this sadly gives no clue about whether it exists or not. The adoption argument is a little bit missaligned in my opinion. Coherence should make an abstract statement about how easily new changes can be integrated, not necessarily how long it takes humanity to do so. While there may be a statistical argument here, a single example might also be influenced by other effects, like trends, etc. Also, just because humans needed a lot of time to end up in a somewhat coherent state of futures, does not imply that there cannot be an abstract pattern here. I think the real question is whether coherence is context dependent or not. Or in other words, can the context influence how entangled the graph looks like.

I would argue that yes, it absolutely can and does. Take biological decoherence on Earth, something I thought about discussing but ultimately cut as I researched more because I did feel out of my depth (and the essay was already long).

Genes are a highly decoherent but relatively stable system. Billions of years to harden, real world constraints to guide them, and good general patterns. Sometimes, the structures that exist within them are quite adaptable and generalized (e.g. the brain), but other times they are very flimsy and fail badly to adapt (see: mass extinction events). Either way, genes themselves are not fast to adapt, only the systems built with genes.

And, importantly, they are _entirely_ dependent on the context of life on Earth, the specific evolutionary paths that we landed on. Silicon based life, if it exists, could potentially have _entirely_ different patterns and strategies that none of the patterns on Earth would be able to address.

On the flip side, you do see _some_ general trends and patterns over time (convergent evolution). The constraints of physics _do_ shape the possibility space, but what works well is contingent on every other aspect of the environment as well.

Same applies to coherence generally. It's always contingent on context.

Reply on news.ycombinator.com