8,345 karma · joined July 29, 2017
People, just like companies, are most likely to act in their own self-interest rather than in the interest of society as a whole. Sure, you'd like to be able to buy everything as cheap as possible, which in a high cost of living (which translates into high cost of manufacture) country like the US would mean relying mostly on tariff-free imports, and the death of the entire industrial base. Is that really what's best for the country?
If the government wants companies to act in a way that benefits the country and its people, then they need to pass laws to make that happen, for example by things like H1B quotas, or by tax penalties that make the desired behavior the most profitable.
I suppose you are suggesting that research mathematicians are either over-qualified mathematically and/or under-qualified in ability to teach, but it seems pretty clear that universities prefer to hire domain experts whose reputations and long publication lists attract students and raise the perceived academic standards of the school.
Your "scenario" of much math research soon being done by AI (supervised and paid for by who, one might wonder), while universities hire people chosen for their teaching skills not academic reputation, seems a bit of a stretch ...
> You're asking why they don't market GPT as "AlphaFold for Mathematicians". They don't because it's not.
No, I am not asking that, nor asking anything for that matter.
I was pointing out that giving a free research tool to mathematicians would likely be better received, and receive better press, than doing something that most top-tier mathematicians are opposed to.
My hypothetical "AlphaMath" certainly could just be free GPT-N access for academics/researchers (as they are also doing to some extent), or it could indeed be a custom system that OpenAI built as a gift to the math community.
As far as the AI companies acting in slimy fashion goes, the most slimy behavior of all is to strongly believe you are doing something that will kill people and cause massive societal disruption ... and still keep doing it.
Got to admire the gung-ho approach I suppose ?
Does it make sense to you for US companies to be sending wages overseas, supporting a foreign family and economy, when there is someone here in the US, unemployed and equally capable of doing the job?
Trump has certainly been ham-fisted about tariffs, but in a world without tariffs then the country with lowest cost of production, closely related to lowest cost of living, and lowest wages, wins, and this is not going to be the US in most cases. If you follow that path then a high cost country like the US will end up utterly reliant on other countries as it imports almost everything, and loses all domestic manufacturing capability. Sounds familiar?
In the short term yes, but in the long term no.
What makes AI different to previous technological shifts is that it is (or will be) a general purpose technology, equally capable of itself doing the new jobs that it creates (so yeah, they'll be created, but also immediately taken away).
The saving grace is that this won't happen immediately, or as soon as we get something the AI companies slap an "AGI" label on in a couple of years, and it will likely take decades until we have an "artificial human" tech that genuinely could do any job (if we choose to allow that future).
Sure - look at what is being done today with H1B visas and tariffs.
What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
Just as with H1B visas, you restrict usage. For example, don't allow AI to replace any job that pays under $250K.
If foreign countries make things cheaper, just as China is making EVs that cost a fraction of a Tesla, then slap tariffs on them.
A government should be primarily concerned about the people it represents, not about the profitability of the companies who are lobbying/bribing it.
Indeed, and the same logic applies as to why allowing companies to replace too many worker's by H1B holders is a bad idea (which equally applies to allowing companies to replace domestic jobs by offshoring).
If you don't hire people, who pay taxes, and take what's left to buy stuff and keep the economy going, then how DOES the economy keep going?
I guess Amodei sees us all on UBI, aka food stamps, so at least the farmers will be selling something, but is OpenAI going to be accepting food stamps to pay for ChatGPT subscriptions? Tesla better be selling it's robots real cheap if it is planning on selling them to people that only have UBI as an income.
For example, how do you reconcile return-to-work mandates with the idea that companies are going to be happy with faceless remote workers? What does the boss do when the shit hits the fan and he would have yelled at people about the need to work all weekend, but instead all he has to yell at is an LLM that tells him he's "right to push back", that it promises not to delete the production database next time (except it will, because it can't learn), and that it could care less about being fired because it's just a calculator?
IMO giving a hypothetical AlphaMath to mathematicians, the same way Google gave AlphaFold to research chemists/biologists, would have resulted in far better PR, and the profit opportunity of attempting to replace the jobs of either research group is minimal.
It's downright bizarre the way companies like Anthropic (primarily), and to a lesser extent OpenAI and anyone else, are themselves pushing the narrative of this tech may kill you, will take all your jobs, etc. That may all happen, unfortunately, but being aware of that possibility you'd think these companies would be a bit more mindful of their messaging and behavior, not just go out with a scorched earth approach of "well, they are going to hate us anyway".
Imagine if they had handled this differently and these results - still using OpenAI models - were coming from the math community. How much better the PR would have been - AI helping math/science rather than yet another story of AI harming society in some way.
No doubt this is what they were at least partially aiming for - not just shooting a trophy animal to brag about, but also being seen to advance math/science, a la AlphaFold, not just take all our jobs and enshittify society with deep fakes and AI slop. But, they heavily misjudged.
In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.
Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.
It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.
It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.
Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.
Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.
I'd have thought they'd be busy (using cached prompt-prefixes) until they were finished.
The 2006 date of this talk precedes his "Surfaces and Essences: Analogy as the Fuel and Fire of Thinking" book where he really makes the claim of analogies being the basis of cognition. I haven't read "Essences", but the major criticism of it is that it was also mostly just endless examples of analogies, not delivering on the claim of the title.
Hofstader's most famous book by far was GEB, which in retrospect seems it achieved it's cult status more for it's sprawling scope and intellectual appeal rather than for any coherent exposition or explanation of its ambitious subject matter. It was a book that felt deep and clever, but really was never more than a bunch of interesting talking points, only loosely related.
Still, having read GEB I had high hopes for his "Fluid Concepts and Creative Analogies" which came 15 years later in 1995, which one might have expected to be equally ambitious in scope and breadth of ideas, but it turned out to be his least popular book, and was more of a narrow discussion on some of his GOFAI programs such as "Copycat" trying to at least demonstrate the use of analogy as a computational mechanism, if not one related to the brain.
I'm reminded now why I never bothered to read his final book "Surfaces and Essences", and reading the reviews of that confirms what anyone who had read "Fluid Concepts", or watched this video, might have suspected - that he was basically out of ideas.
For anyone interested in analogies and their role in language and thought, a book that has better stood the test of time is George Lakoff's "Metaphors We Live By".
Presumably the people that OpenAI and Anthropic are trying to impress with these trophy kills are potential IPO investors, but I would have thought investors would also be concerned about the growing public backlash against AI.
Clearly the real-world cannot "blow-up" - real-world water vortices do not reach infinite velocity, etc.
The point of having Navier-Stokes as a Millennium prize was to hopefully generate new mathematics and techniques along the way, and auto-generating a sprawling AI-slop proof or millions of lines of Lean does not accomplish that result.
Clearly OpenAI has no interest in the math itself - to them this was just a trophy animal to shoot and stuff. I would be very surprised if they now helped analyze the proof and try to extract the mathematical value out of it, and this would obviously require outside help who are probably not inclined to help OpenAI math-wash their behavior.
Just stop doing that. Don't treat unsolved math problems as some cheap benchmark to beat.
Leave the math for mathematicians, and let them use AI in a way that helps the field, not in a way that harms it.
25 Fields medallists! Wow!
Our cortex is connected in a loop with our thalamus (cf recent talk of looped transformers!), which can be separated into thalamic core and matrix. Basically our senses feed into the thalamic core, and cortical outputs (predictions) feed into the thalamic matrix. The thalamic connections/loop seem to act a bit like a patch panel connecting different regions of the cortex.
The cortical microcircuit is a real physical circuit - a pattern of synaptic connections between layers and inputs/outputs. Those Transformer circuits are more functional than physical. We know how a Transformer is physically connected, with attention heads etc, and this "circuit" analysis then describes how it functions, e.g. with attention heads in adjacent transformer layers effectively combining to form "induction heads" that copy data across embeddings.
Perhaps the common occurrence of analogies in language is because language is distinct from the things it is referencing, so some degree of abstraction is unavoidable?
IANAL, and I'd be surprised to see any lawsuit come out of this, but you certainly don't need to have "violated the law" to be on the receiving end of a lawsuit.
- The never-same-twice nature of sensory perception
- The need to recognize objects from different directions, distances, partially obscured, etc
- Because the nature of the world is that predictive categories are equally or more important than individual identities, especially to basic survival, and for basic organisms
Rather than "generalization" being unspecific, I'd say that as far as perception goes there is a very concrete realization of generalization in our brain - embeddings and associative recall by partial embeddings (see half a face, recall the whole face).
Analogies don't seem to offer any benefit over prediction in terms of what to focus on - there is always a need to separate foreground (object) from background, and to learn what is instance-specific vs generalizable. Is is irrelevant or coincidence that sabre-tooths are near the water hole, or is this part of a predictive pattern (or analogy of some sort?!) that should usefully be learnt?
When we are not sure about what we are looking at, we often then switch from the gestalt "looks familiar" to looking for specific points of confirmation. Is that car in the parking lot really ours? Does it have our air-freshener hanging from the mirror?
Needing to focus on details can also be part of generalization and exception making - when we recognize that two situations that seem similar are in fact different, and we are trying to isolate what the differences are.
I think that the strongest argument for our brains being built for prediction, not analogy making, are that:
1) Prediction is exactly what is needed to survive and thrive - to take advantage of the external world, and not be killed by it. We directly need to predict. We do not need to make analogies, unless analogies can also be used as a way to predict.
2) Our brain appears to be built for prediction, and prediction gives rise to the very obvious learning signal of prediction failure. One of the lessons of LLMs is confirmation of the power of prediction as all you need to drive learning.
The overall way our perception seems to work is indeed as much by prediction as sensory input - in a way prediction really comes first, with sensory inputs narrowing/correcting the prediction (and when you turn it off then we predict/dream without the constraint of sensory reality). The environment you are in already leads you to predict what might be there, and memory of what specifically was there previously will shape those predictions. Sensory feedback corrects/narrows those predictions, and then - if necessary - we look for specific memorized features to confirm what we are seeing/sensing.
Considerations of evolution, and what would be the sensory and cognitive "needs" of organisms of ever increasing complexity, all the way up to ourselves, can also point to what brain capabilities are older and more fundamental, and would have been there to be refined, adapted and built upon later. Even the most simple organism needs to never-same-twice generalize, but will be stuck being reactive until evolution has either equipped it with "reactive predictions" such as the fly being startled by the fast-moving visual input (predictive of being swatted by the horse's tail), or eventually developing a predictive brain like our own, especially useful to an animal that is a generalist living in an environment that is more difficult to predict.
If someone said to you "you're going to regret this, mo-fo!", do you really need to suppose they had a specific plan in mind, rather than just intent to intimidate?
If you want to suppose that Brubeck had a concrete plan for how he would ruin Buskmaster's career, then my best guess would be that he was threatening for OpenAI to publish without giving any credit to Buckmaster (who has been working on Euler overall for at least 10 years). Obviously this would be absurd, but no more absurd than what Brubeck was also demanding - that Levant was excluded for any credit and could not have his name on any OpenAI writeup (with this being the exact sticking point that Buckmaster was refusing to accept).
The funny thing is that this threat, as is often the case, seems to say more about the insecurities of the person making the threat than the person they are directing it to.
It's notable that when we do use analogies there are really two use cases
1) Trying to use them to communicate an alternate or simpler POV
2) To make a joke, by pointing out a ridiculous comparison
The joke case is interesting, because this seems to be the basis of half of humor - breaking expectations (the other half perhaps being taboo breaking?), which really isn't how you'd want an adaptive cognitive system to work. You need to see the predictive regularities in the world, not see absurd similarities that make you laugh.
Our cortex (all of it - visual, auditory, association. motor..) is remarkably simple and regular - it's six layers of neurons with a specific circuit design, that appears to make it a predictor, with top down predictions meeting bottom up sensory feedback. As noted, it makes a ton of sense that evolution would have equipped us with a predictor (and the more powerful the better). Prediction directly supports reasoning and planning (multi-step what-if prediction).
So, my question for Hofstader, or you, would be given that the brain seems to be literally built for prediction, then what is to be gained by regarding this as analogy making instead? Is there any evidence for this? Even as a philosophical viewpoint, doesn't that make for very convoluted explanations of simple things like reasoning and language? (I explained prediction-based reasoning above in one sentence - the analogy-based explanation takes a whole dense book!).
One could certainly use the language of analogies to describe perception (Pooh's ears on the towel are analogous to his ears in the book, etc), but what is the benefit of doing so, rather than the simpler view of this just being generalization (which is specifically what the animal needs), and where is the evidence in the brain?
It's comparable to Magnus Carlson saying that if he wanted to cheat, all he would need would be for someone to tell him to spend more time thinking about a specific move (just a wink would be enough) as an indication that a computer had found something interesting.
It's as-if after OpenAI first failing on Navier-Stokes (which OpenAI had just tweeted about 2 days earlier!), someone winked at them and said "you might want to try a little harder ...".
Terrance Tao has lamented this practice as being unhelpful for mathematics, and likely to lead to humans working in private to avoid this.
Tao has also noted that many of these AI math proofs don't really help mathematics (nor does it seem they are intended to), since for many of them the proof was never the point, it was the math expected to be needed to be developed along the way, which the AI solutions don't provide.
Given Buckmaster's telling, this seems beyond "poor choice of words"... It was a veiled threat, that he then doubled down on with his "If you don’t want me to be nice, then I don’t have to be nice." follow-up.
**
I said that if OpenAI released its result in the way proposed I would go public with what happened. The reply was, “Why would you ruin your career?” I replied that I am an academic, and asked why he thought going public would ruin my career. The reply was, “If you don’t want me to be nice, then I don’t have to be nice.”
**
FWIW there are also other people on Twitter, such as this DeepMind researcher, saying this is a pattern for Brubeck.
https://x.com/dheeraj_nagaraj/status/2097266146445774924?s=2...