"Battle not with monsters, lest ye become a monster, and if you gaze into the abyss, the abyss gazes also into you."

There is nothing scarier than to think that next word predictor can be sentient, because it means we are just next word predictors, and all our pain and suffering is beyond meaningless.

In the same time I think there are definitions for AGI that can fuel exponential development in all important fields (from protein folding to medicine to enegry and etc), and this can just as well be a transformer large enough that has reasoning.