I think something that has concerned me with the concept of scaling to AGI is the concept of "adversarial examples". Small tweaks that can be made to cause unpredictable behavior in the system. At a high level these are caused by unexpected paths in high dimensional model weight space that don't align with our intuition. This problem in general seems to get worse as the size of the weights grow.
From a value perspective a very high fidelity model with extremely unexpected behavior seems really low value since you need a human there full time to make sure that the model doesn't go haywire that 1-5% of the time