>>It seems like quite a paradox to build something but to not know how it actually works and yet it works. This doesn't seem to happen very often in classical programming, does it?

Well, it is meant to be "unknowable" -- and all the people involved are certainly aware of that -- since it is known that one is dealing with the *emergent behavior* computing 'paradigm', where complex behaviors arise from simple interactions among components [data], often in nonlinear or unpredictable ways. In these systems, the behavior of the whole system cannot always be predicted from the behavior of individual parts, as opposed to the Traditional Approach, based on well-defined algorithms and deterministic steps.

I think the Anthropic piece is illustrating it for the sake of the general discussion.