>It's possible that we're already roughly as intelligent as it's possible to be.
Now that's more depressing than global annihilation.
Now that's more depressing than global annihilation.
For many 1000's of years different religions, some on opposite sides of the world from each other, have been saying the same thing. We have the ability to be better people, but all of us squander it with pettiness, jealousy, greed etc.
Fair point. But I honestly wonder if it's not true, or close to true. I mean, people have been talking about the "end of science" for a while now, even though we know we're not quite literally at "the end". But let's say that we unify relativity / gravity / QM in the next 100 years or so, and identify the exact nature of dark energy / dark matter. Presumably that would represent knowing close to everything about the physical world that its possible to know. And if we can discover that, it does lead you to wonder "what else would more intelligence represent"?
Of course, maybe we're WAY more than 100 years away from those things. Or maybe we'll never get there. Or maybe if we do, it doesn't mean anything vis-a-vis the limits of intelligence. I'm just thinking out loud here, this isn't an argument that I've put a lot of time into developing...
That's a severe misunderstanding of what it means to have a unified physical theory. The current standard model of physics allows us to theoretically predict the evolution of physical systems with an insanely high precision. A precision so great that there wouldn't be any meaningful error simulating macroscopic objects like human brains, given sufficient computing power.
But we still have massive holes in our understanding of many things, including human biology or neurology. We don't have enough computing power to precisely simulate, but even if we had that power, it wouldn't necessarily let us understand anything. Elementary particles and humans are on hugely different scales. Observing a simulated human would be easier than observing a human simulated by physics itself, but it wouldn't just magically make everything clear.
Having a TOE in physics brings us very little relative benefit compared to the Standard Model, when it comes to understanding biology, materials science, geology, psychology, computer science, etc.
Sure, like I said, I'm just thinking out loud here. Obviously TOE in and of itself doesn't automatically translate into full knowledge of everything in practice. But in principle, it would given sufficient computing power, allow us to simulate anything, which I think would yield additional understanding. But here's the thing, and where this all ties back together: sufficient computing resources may not be possible, even in principle, to allow use of that TOE to do $X. But an upper bound on the amount of computing resources should also reflect something of an upper bound on what our hypothetical AI can do as well.
Or to put it all slightly differently.. if we had a TOE, we'd have shown, at least, that humans are smart enough to develop a TOE. Which I think at least raises the issue of "how much smarter can a machine be, or what would it mean for a machine to be much smarter than that?"
Note to that I'm not necessarily arguing for this position. It's more of a thought experiment or discussion point than something I'm firmly convinced of.