1,364 karma · joined February 23, 2014
And dealing with those pathways individually would be far more productive than attempting to control the proliferation of algorithms that think, which in the long term is probably impossible. It also leaves us on far firmer scientific footing. An individual pathway - bioweapons, nuclear weapons, etc. - is far easier to reason about and accept/reject a notion of feasibility. Treating an ASI as a machine god and asserting futility in face of that god is not going to get anything done.
No one serious is really arguing for a scenario where the machines rise up a la Planet of the Apes. The dangers people are really examining are scenarios of either one exceptionally clever innovation: a designer virus or hacking NORAD; or cleverly amassing economic / political influence over time akin to an exceptionally clever tech magnate. And these roles could adequately be filled by either ASI or a Bond villain.
If this line of thinking is taken too literally, we can never falsify it. Any specific hypothesis - nukes, bioweapons, spontaneously convincing us that life isn't worth living - can be deflected with the objection that if we can anticipate it and prevent it, it is not the route for a true ASI extinction event.
It has proven surprisingly resistant to AI so far - Astra can make a very quick prototype but upon review it had many defects. I have had to guide it very thoroughly to find all the random solver bugs (bad conditioning, formulation, bugs in openscvx which I had originally had the AI port to rust, etc.) preventing well-behaved solves. But I think (hope?) I have turned the corner on these.
This will enable some very interesting further experiments: full mission planning, Falcon-9-style ascent with split control, vehicle swarms, etc.
- OpenAI lying their face off for marketing purposes
- Buckmaster sore about getting beaten to NS, misrepresenting what was told
- Buckmaster not being an expert in ML, not understanding what he was told and hearing what he wanted to hear as it vindicated him
- Others...
OAI claimed that none of the people on their team were domain experts. So it's more akin to Deep Blue than Stockfish (actually slightly better than Deep Blue which did hire domain experts) but nevertheless represents a highly sophisticated automatic theorem prover.
https://xcancel.com/SebastienBubeck/status/20973794116915163...
"If in other sciences we should arrive at certainty without doubt and truth without error, it behooves us to place the foundations of knowledge in mathematics."
> For example, when Timothy B. Lee looked at a spreadsheet that Zitron used to create a projection of Anthropic's revenue, he found, "He doesn't count February 1-10, counts March 1-10 twice, counts August 21-October 21 as one month instead of two, and doesn't count October 21-November 1. [another commenter notes that his spreadsheet also contains February 30] ... Ed claims he tried to compute Anthropic's revenue for 2025 and came up with $3.6 billion, suggesting some funny business [but the numbers work out once you fix the errors]"