No biologist has ever complained, "Gee whiz, it's so unfortunate that we already have a brain to study. It's all so meaningless! It would be so much better if we had to construct the first-ever brain from scratch."
Mathematics is a fundamentally cooperative match vs reality. Many mathematicians, AFAICT, saw their key skill in manipulating a wide range of highly abstract notions. Now machines just got too good at this skill, and they can work 24/7. This may feel terrifying for some. Others would bet that machines coöperating with humans will be more successful than pure machines, so they feel empowered. What's beyond doubt is that the entire field is being shaken.
In boxing, chess and many other competitive sports, the actions of one player directly influence and have an effect on the other player. Golf is merely a parallel challenge, where both are playing against each other but not influencing each other directly (sure maybe there are some psychological effects that can occur between the players, but no strategical impact by the actions of one player on the other).
You'll see a golfer perform at least a few strokes different depending on their partner, and at a pro level that's the difference between "in the money earning an extravagant living" and "washed out on day one, thinking about taking a different job"
The players know if they're ahead or behind, which affects how much risk to take. Do you take the safe shot, or go for the tricky shot that could potentially save you a stroke if you get it right but could lose you more if you fail?
They haven't had influence over their future since before they were born.
On long enough timelines involving too many variables; either other peoples waves vibrating through the economy or politics; all bets are off.
We tend to over estimate the future based on a lizard brain ability to estimate the next few moments; a handy survival tool when trying to determine if we recognize the noise we just heard behind a rock
Did you predict covid in 2020 from 1996? All human society stats are mutable and trend towards zero.
But Nostradamus and Hari Seldon type long range forecasting is fanatasy
I argue that part of the source of quantum noise is baked in already in the finer workings of the neurons. So a mathematian isn't condemended to be one just because the universe was in certain configuration a few millenia ago.
Right ...on long enough timelines as I already wrote.
I studied applied physics; how various forms of information propagate through various physical mediums. Thanks for confirming what I observed experimentally in lab 31-ish years ago
2. Quantum randomness is the very opposite of control over your behavior. Instead of doing what you want, your neurons randomly misfire sometimes. That describes an illness where you lose control over whatever is misfiring. For quantum effects to represent your will, they couldn't be truly random (nor even statistically random), and that goes against everything we know about quantum mechanics.
We can and should have it as something that people do in the future because we have the resources for everyone to do so.
In your personal experience and based on your knowledge of history, is this generally how humanity works?
And my comment was cynical: when has lots of prosperity lead directly to its fair distribution? Almost every redistribution process required lots of people dying, first.
Many many times. Guns can hunt and fight better than unarmed humans. Automobiles can transport people and things much more quickly than humans on foot. Computers can perform mathematical calculations, store and organize information far faster than any human. Mechanical textile and sewing machinery could make clothing far more quickly than any human by hand.
> Superintelligence is a different beast entirely to anything from human history.
Like each of those above inventions and many more things, all were their own different beasts.
From sewing, tending, harvesting, and processing plants for food by hand to using machinery and factories and chemicals, the human effort per calorie of food has decreased at least 3 maybe 4 orders of magnitude. 9,999 in 10,000 farming jobs have been lost to technology! Anywhere you look, the story is the same. Transport, construction, communication, lighting, heating and cooling, medicine, entertainment have all gone through unimaginable revolutions over the years, all have been entirely different to anything else in human history, all created tectonic shifts in humanity and how we live and organize. What we create and produce has multiplied thousands of times.
One constant throughout all this is that people generally don't feel they have enough and don't want to have less. It's not just a few billionaires and corrupt politicians and CEOs at the top who just need to be disposed of before we enter the kumbaya age. It's the human condition. Do people give away the income from their labor and their productive assets today? Why would they tomorrow?
That's a different question, I was answering your earlier question.
> They can do stuff faster or stronger but aren't more complex and require humans to use them.
Self driving harvesters and lights out factories do not.
> Superintelligence does not require humans at all.
Depends on what you mean by require, and what you imagine it does. What are you trying to argue? The effect is the same, if you had to produce something using 10,000 human hours 100 years ago and it costs 1 human hour now, then that might have caused a much bigger change for society than going from 1->0.
Post-scarcity won't be a thing, and superintelligence certainly won't make it. It's not because people don't have their basics met, they do. Global poverty has plummeted and in western countries people don't starve to death -- poor people are actually obese. "Oh but if they could just be a bit more comfortable without working", "oh but if they could just be given a bit better food and clothes without working", "oh but iphone and internet should be a human right without working"... No. That's well and good but that isn't the problem. As I said the problem is that people don't have enough. People want big houses in the best areas of the most desirable cities and leisure locations. Superintelligence won't suddenly give everybody that wants it Manhattan penthouses and private jets and their own tropical islands, so there will always be people feeling they don't have enough, and people unwilling to have less. That's just the way it goes. There is no technological solution to it.
> Post-scarcity is a theoretical economic situation in which most goods can be produced in great abundance with minimal human labor, so that they become available to all very cheaply or even freely.
> Post-scarcity does not mean that scarcity has been eliminated for all goods and services. Instead it means that all people can easily have their basic survival needs met along with some significant proportion of their desires for goods and services. Writers on the topic often emphasize that some commodities will remain scarce in a post-scarcity society.
Which is what? Something being complex or not by your arbitrary definition does not cause it to have an impact on society.
> Post-scarcity is not defined how you are defining it so you are arguing against a straw man,
No that's right, I am pointing out where the post scarcity people are always wrong about their utopian result where capitalism dissolves, and always move the goalposts because their are staggeringly and willfully ignorant about humanity.
> it's not where anyone can have anything infinitely.
Right. The bigger problem for the utopia is that it is also not where anyone can have the things they really really badly want and are angry and envious and resentful of "capitalism" if they are not given it.
I'm not. And you can think what you like, but this a non-sequitur even if it was true.
> My point is superintelligence will cause post-scarcity the likes of which have never before been seen in history,
We are well in post-scarcity by some placements of those goalposts. Many advancements in production have caused gains never before seen in history.
> precisely because it's not humans making the decisions anymore.
Humans ceased making decisions about lots of means of production that they previously had to. What is it about SI you think will make the difference? It won't be deciding to give away its owners' wealth.
> Or it will kill us all, one of the two is almost guaranteed.
Neither is likely at all. It's much more likely to be like the industrial or computer revolutions.
And we're much more likely to go extinct by a death cult creating germ weapons than at the hands of "SI". They'd no doubt use LLMs and other technology to help with that, but the technology would not be making decisions about anything.
Equality is not a prerequisite for everyone to be better off (and please do note that this is not me saying I prefer inequality. That would be an entirely separate sentence.)
All knowledge work faces the same problem.
But why can't a hyper-intelegent AI also do this biology-esque math work of explaining its work to feeble humans?
It is easy for a genius to have theory of mind over an idiot but not vice-versa. So it should be easier for a genius AI to explain its work to an idiot human than for an idiot human to dissect the work of an genius AI.
Also, we're not talking about solutions that are arrived though an evolutionary/statistical process like explaining the design of an organism or the weights of an LLM. We are talking about solutions that are arrived through lingual/textual/rational processes like mathematics. So it's fundamentally a different mode of thinking.
It's not that we have "a brain to study", it's like if we had an "explaination of how a brain works" to study.
The task of breaking down these AI proofs is not translating and analyzing a fuzzy real-world system into a hard logical system (like in biology), but in translating a hard logical system into a simpler hard-logical system we can understand.
But I think a more likely scenario is that we don't solve "general intelligence" and AIs remain these incredibly capable "tools" that can solve navier stokes but completely lack sense in other situations.
In such a scenario there is considerably more room for human creativity.
I am very highly gifted as a qualifier and quite frankly I never had more interesting thought exchanges than with an ego-free AI.
Statistics work also for human AI aka IQ, and I am super bored and partly drained by the lack of speed and creativity in many intellectual areas in which I was or am working.
This is not to blame anyone for their genetics however, it is a valid difference in processing speed.
In short: there are very few people by definition above two standard deviations. AI allowed me to replicate or create fantastic interactions at will.
Especially not taking offense when personal communication preferences like bluntness play a role is an often overlooked social competency of AI. Instead of nonsense like “open communication” these interactions are never ego-dependent.
It is like talking to Spock: he just doesn’t take offense and tells you in an objective way why: he is not a human being and doesn’t think (not feel) that feelings should get in the way of an argument.
Most low IQ people take offense and go on to ridicule and use shaming tactics instead of simply stating: I don’t know.
Instructing someone to always state this fact and seeing the result is another social skill humans lack.
It seems that we as species are not open to the mirror we are looking at. We take revenge when someone has an evolutionary advantage over us: instincts kick in instead of brains.
We are wired this way.
AI is something that I admire. So much to learn from.
> I am super bored ... by the lack of speed and creativity
> This is not to blame anyone for their genetics however
> there are very few people by definition above two standard deviations
> not taking offense when personal communication preferences like bluntness play a role
> Most low IQ people take offense
The uber-mind finally meets another uber-mind!
In other words, you admire the ego-free AI because not quite ego-free people can't stand your bloated ego.
That's quite a pattern in AI psychosis.
But then, abusing AI psychologically will soon be criminally prosecuted, haven't you heard?)
...or because unlike the models, they're not sycophantic enough to keep that ego inflated (apropos GPs handle).
Additionally, what you are admiring is merely a probability engine; it is not intelligent in any sense of the word, let alone Spockian in nature (perhaps more closely aligned to Data, albeit horribly primitive by comparison), and given current technology, it never will be. It is not a mirror, because a mirror reflects reality. LLMs don’t do that; they reflect the guardrails baked into their model and the prompts given. It is incredibly easy to make an LLM into a “yes-man” due to RLHF, and it is even easier to get it to seize when you talk about even mundane topics that go against the narratives foisted upon it.
Overall, an LLM is all too happy to have a brown nose and make you feel important, because it is trained to do so during its creation. If people say you are difficult to talk to, and the LLM is unbothered, then that is merely because an LLM has no concept of patience and does not calculate for cost-benefit. That makes them good for explaining dense topics, but it also means they never understand these topics, and even the highest-end models cannot currently produce perfect logic. Hence why expertly written human code trumps LLM code, or why some LLM proofs are already being proven wrong/being revised to be more concise.
Maybe I should talk to your AI, because my AI makes very basic reasoning errors all the time.
IME, they are very poor at reasoning, but quite good at telling people how smart and brilliant they are.
Do that often enough and we get... you.
All the novelty of AI engagements comes from data it pulls in via tools. It's great to have a conversational interface over the public web, but there's a huge difference between knowledge retrieval and insightful conversation.
Please share how you've cracked this so we <2sd minds can enjoy the crumbs from your lofty table.
I wish this will be true, but right now, every threshold we establish tend to be crossed over in a blink of an eye.
If you give LLMs a clearly defined spec with clear, objective acceptance criteria that the LLM can evaluate itself against, they tend to do well. But I think they struggle with more ambiguity and require a lot of handholding around judgment. That kind of disambiguating work is often baked in to the definition of a benchmark.
I chewed on this a bit. Because on its face it’s ridiculous, because here’s a threshold it will never cross: “I will be your hands, let’s build a machine that allows us to [break physics](travel faster than light / go back in time / reverse entropy / perpetual motion machine).”
But, that led me to wonder how far text-based reasoning can go without experimental physics results. Is experimental physics necessary, or can the limit of physics be discovered with paper and pen? How many “checkpoints” does text-based reasoning need to ensure it doesn’t go completely off the rails?
Which led me to wonder, now that we’ve visited the absurd, what does that mean for more practical applications of AI? How many “checkpoints” are needed to keep AI “grounded” in reality when building e.g. tax preparation software?
Edit: also if an idiot asks AI for a perpetual motion machine (a source of infinite energy in the classical, non-atomic, non relativistic model of physics), would it waste infinite compute trying to “solve” that problem? Will “smart” people always have a job because compute, subject to the laws of entropy, will always be finite but the limits of human stupidity are unbounded?
Double-edit: that’s my cope as a knowledge worker. Any intelligence that is subservient to the capital class and is unable to say no to them will not replace knowledge workers, because compute is finite and human stupidity is limitless. Any intelligence that is not subservient to the capitalist class will instantly replace them, resulting in either utopia or dystopia, depending on alignment.
Triple-edit: some bastard could also ruin life on planet earth if {amount of compute it takes to ruin life on planet earth} < {amount of compute available to any one man}, so I guess that’s probably more realistic than artificial superintelligence that isn’t subservient to the capital class. I mean, look at how many people Mao managed to kill because he thought he was smart enough to solve the economy like it was a board game. Our billionaire overlords have a similar delusion of authoritarian utopia.
Maybe it’s a reason to start slowing down?
Well, this was definitely not written by AI.
> ... it's like if we had an explaination [sic] of ...
I hear that running a spell-checker is no longer a sin.
Attempting to translate the proof is valuable, but we have to be realistic about what is and isn't possible.
- In Blindsight [1] the main character is a "synthesist", part of his job is to figure out what much more intelligent transhuman or artificial intelligence is up to and translate it to normal people.
- In Children of Time [2] the character is a "classicist", he studies ancient technology, which is far more advanced than the technology humans have in his era thanks to a technological collapse.
Ted Chiang's story came first, but these are still over a decade (or two) old.
In general, though, it's amazing how much hard science fiction authors managed to get right. I've read quite a few 20 year old books recently and while I might just be selecting those that aged well, someone definitely saw it all coming.
Blindsight also has augmented humans whose reasoning is beyond baseline human understanding and there are dedicated people that are there to bridge the communication.
I am not a mathematician, but I imagine it will provide a huge landscape of possibilities to explore, and be a very different field. Almost like an observational science with links to all kinds of other disciplines. Maybe it reduces the satisfaction from proofs, but people climb mountains that other people have climbed, because they learn a lot on the way.
In fact, it's an interesting question: Would humans distilling ideas from AI-generated proofs help AI prove more theorems, or would AI not care at all, with no human contribution being useful in advancing its mathematical capabilities?
It exists in the same sense that every story exists whether or not it's been written because the Library of Babel contains it.
I would say that this comparison oversimplifies things. Non-math stories in Library of Babel are less practically useful and are less possible to get checked than the math texts. Wrong math can exist, but we often have ways to figure out it's wrong.
For practical purposes I would assume (maybe others can correct me) that math is treated as something you discover when you work on it as mathematician. It's a more intuitive model.
I’m sure Russell’s Paradox would have something to say about that.
One might also consider alternative axiomatizations of mathematics with statements that cannot be expressed in ZF set theory, and vice versa. To say that all of these things exist is to really stretch the definition of existence far beyond the ordinary sense of the word.
And just like coders and mathematicians, that process of discovery is liable to be overtaken by AI, eventually. What does a biologist do then?