But given the exponentially greater chemical, biological, and neurological complexity of human brains, the millions of years of evolutionary "pre-training" that they get imbued with, and the years of constant multimodal sensory input required for their culmination in what's called human intelligence, it takes an extremely bold assumption to believe that equivalent ends can be met with a stream of fmadds driving through in an array of transistors.
It's hard to overstate how big a gulf there is in both the hardware and software between living systems and what we currently see in our silicon projects. Neural network learning in general and LLM's in particular are like crude paper airplanes compared to the flight capabilities of a dextrous bird. You can kinda squint and see that they both move some distance through the air, and even marvel at it for a bit, but there's a long long long long way to go.
Yeah, nature does it bottom-up through evolution of self-assembling, self-replicating machines. A lot of "design constraints" that go into living things has to do with... keeping them alive. Compare an electrical wire or silicon trace with a biological nerve or neuron - the former are just simple traces of inert metal, the latter are complex nanomachines mostly dedicated to self-maintenance, and incidentally conducting electricity.
Point being, we can't compete with nature across every dimension simultaneously, but we don't need to, and we shouldn't, because we don't need most of the features of natural solution, not at this point. We don't need Concordes hatching from eggs - we have factories that build them.
However I concur that we may end up creating an AI which thinks nothing like us, has no sense of morality, and yet far exceeds us in intelligence. And that would be somewhat unsettling.
We may build something that far exceeds us in capability, but without us understanding it, or it understanding us. This is the alignment issue.