jrowen

Having 10,000 (or more) agents working for months on a problem is categorically different from the type of usage most people are familiar with. The complexity explodes to a point that requires the "archaeology" he mentions for people to even try to understand what's going on.

This is the key insight. Those capabilities are currently limited to frontier labs and literally cost them millions of dollars to achieve. But as we well know, computing costs and algorithms are optimized rapidly and soon those capabilities will be more widespread. We are at an inflection point where we can design systems that are able to go and do things that we don't fully understand, and they will handily outpace our forensic bandwidth.

*https://en.wikipedia.org/wiki/IceCube_Neutrino_Observatory#/...

I watched a 4 hour podcast from the excellent "From First Principles" guys* that affected my thinking on this. For the first time I feel like we actually are on that precipice.

It differs from the nuclear bomb in that it will be harder to regulate and it will be incredibly difficult to keep out of reach of bad actors. It's just code and data. Somebody is gonna take that code and make the next version of it. Lots of people, in fact. Compute resources improve and proliferate orders of magnitude faster than uranium enrichment facilities.

But I think those technologies are also linked in that the only surefire scenario I can envision where AI kills us all before we can pull the plug is launching nukes. So in that more technical sense let's just start by making sure they can't do that. The killer virus epidemic is another one to watch, and they touch on Anthropic's biolab in the pod. That one seems less instantaneously existential, but also more distributed.

It's basically going to be about...how do we secure everything in the presence of a swarm of actors with poorly-understood capabilities? We kinda have to disable and disconnect and just remove things that can be used against us from the equation (a la BSG).

* https://www.youtube.com/watch?v=NGfGw1tGxUY&t=3s

I don't think "just make it illegal" is going to save us, that doesn't make me feel safe anyway. They may try that first because it's easy - create a regulatory body, sign some legislation, problem solved! [george-bush-mission-accomplished.jpg] But at this point I feel like some kind of Battlestar Galactica scenario is most likely - hopefully not quite as existential - but it will take a collective reaction to a traumatic event (a la Hiroshima/Nagasaki). Technical rather than (or in addition to) legal measures will be taken, like network partitioning and hardening. This is everyone's problem whether we like it or not.

(I'm not saying this "fatalism" should be used as an excuse by anyone working for any of these companies, it should give them pause that any bloodshed would still be squarely on their hands, but as an observer, people are gonna keep pushing until shit hits the fan. [jeff-goldblum-jurassic-park.gif] It's also not really about whether it's "appealing" or not, it's just trying to predict and anticipate different likelihoods...)

It sounds like you were rather negligent as well. You didn't care to have any inkling of memory of the vesting schedule in 1996, or just to double check and "wrap up" the financial details of that agreement after the work was done?

It does seem somewhat suspect to elide mention of what became of the shares that you did own, and that you only re-noticed Nvidia in 2024. Every 14 year old PC nerd/gamer knew that name in 2001. It is quite a feat of negligence to wait that long to dig this up.

You seem like a decent person, and I do believe that you are justly owed something, but I must admit that I find this to be a depressing tale of rich people spinning wheels for naught. A modest inflation-adjusted multiple of the original value of the options seems reasonable.

Before anyone accuses me of shilling or white-knighting for a major corporation, I don't have any broader sympathy for them and I would generally take the side of David over Goliath, which I think some commenters are making this out to be, but it seems more like Mini-Goliath and Mega-Goliath are bikeshedding and David has no stake in the matter.

This is true, but, in their efforts to maximize short term gains, they are moving us towards your "true globalization" whether they intend to or not. That's kind of the inevitability. You can't send all that economic activity and expertise over there without lifting those people up. They are inexorably chipping away at the reserves of cheap labor to arbitrage. China's getting better, and I'm here for it.

It's called being charitable. Trying to understand what their point is and responding to that for the sake of advancing the conversation, rather than grinding it to a halt with pedantic quibbles over inaccuracies or poor phrasing.

I've never been so I can't really comment on the atmosphere of the building.

You made a pretty specific point and prediction about the price of tickets, which I responded to, but now you're not talking about affordability. Because the suggestion that they might not be doing the most soulless oligarchic thing ever in this place doesn't fit your doom narrative.

This whole subthread is from like an alternate timeline where the world actually found out or agreed that globalization was bad and stopped or slowed it somehow? Did the US meaningfully take back manufacturing or something?

If globalization is not inevitable, what's the alternative?

Perhaps I'm looking at this from a too zoomed-out or detached perspective. I'm just thinking about the concept of the technology at play and its future impacts regardless of who owns or controls them. But maybe that's naive. In today's connected world you can't separate those things, and I think I'm understanding that a lot of the outrage here is more about the socioeconomic implications of a big corporation suddenly "disrupting" a field that was largely owned by more altruistic academic institutions.

I do still think that the internet, despite largely being owned and controlled by the powers that be, has represented a great shift of power and agency to the individual, and is ultimately a socialist construct, and we will continue to see those changes over time, and that AI is an extension of that.

If there's a distinction being made between "using AI for math" and "the particular methods being employed by OpenAI", I think that's splitting hairs. The core of the issue is that math is being done outside of people's heads by a different thinking system and they're worried about the implications.

If you look at something like the EPA (or the FDA as you mention), it is true that certain uses and side-effects of technology have been identified to be detrimental to society and have been restricted, but this letter doesn't seem to be in that league, it really seems like sour grapes to me. Dumping toxic chemicals into the water table is one thing, saying "we don't think people will learn from these methods in the same way that they traditionally have" doesn't hold as much water to me.

Nobody is going to care that the math isn't being done in the traditional way. The results speak for themselves, this is now a part of the landscape. No amount of hand-wringing is going to put the cat back in the bag. Adapt or perish.

Everything is sensationalized, every super niche happenstance is sold as earth-shattering drama, the outrage arms race is so tiresome.

The cops will find a way, the criminals will find a way. It's ultimately people v. people with access to the same level of technology.

It's somewhat sad to me that this phenomenon is borne out in the comments (of this thread and, I would say, generally across HN in my experience). It's so easy to prophecy doom, ignoring the fact that life and humanity has generally prevailed and continued "progress" in the face of adversity, evil, extinction events, and other setbacks.

I'm generally in agreement with this article, although, I'm not sure I'm onboard with the dismissal of utopia. Utopia can be whatever you want it to be, it doesn't have to be defined as impossible nonsense. If we're in a protopia, and we want to see protopia, there's sort of a tautological nothingness, akin to pointing out that there's no such thing as free will. The message then boils down to "well just keep on truckin' as is."

The Millenium Scholarship program in Nevada paid for tons of kids' education by taxing casinos. It's not perfect but it's channeling economic activity to the people.

I once had an idea for a system that basically took in multiple video inputs and produced video output. Individuals would then run instances (that they could tweak) that were connected to a larger network. Somewhere between an MMO and Folding@home. Not only would it distribute the compute load but it would simulate the chaos of large numbers of individuals trying out different things.

Again I think it comes back to the fitness function and how things get nudged "upward." In any case it seems all you need to do is get fitness right and then let it rip...

Have you explored any distributed concepts like that?

Maybe there is the possibility of some kind of stack? Something that starts low level and somehow can be composed or integrated into higher levels of abstraction?

I don't understand this. It's the endgame. Compromise for what? Working an invented job they don't actually want to work? What is your idea of a solution that combats misery better than a free paycheck?

People are confused into voting against their own interests all the time. Socialism is a bogeyman but Social Security is great (if UBI is too odious just call it lowering the social security age).

I'm to be seen as "you filthy masses" guy when you seem to be advocating some form of mass violence? You could actually, just, exercise democracy to redistribute the wealth. I daresay that would be smarter.

Obviously, you have to "cheat" biology somewhere, we don't have hundreds of millions of years. Neural networks cheat by essentially throwing out the whole evolutionary process and environment that led to the brain, attempting to make a model that works like the brain. IMO that is too much cheating.

Starting with a substrate of random bits of assembly code strikes me as a little too low-level (which is not to dismiss this research at all, I think it's valuable, I'm just spitballing big-picture ideas). Have you considered starting with something like the Unreal or Unity engine, or even Minecraft?

You would lose the elegant and unopinionated search space of all programs, and you would have to engineer a more structured system (kind of like Spore but more simulation than game), but I feel like you might be able to get some more readily relatable behaviors sooner?

It's like when Facebook was the bogeyman. Facebook is just a mirror. If you see bad stuff happening (e.g. people spreading and engaging with misinformation), that's the people, not the platform. Unless one is claiming specific intent on their part, Facebook (like most tech) enables good and bad in equal measure.

Getting rid of, or pushing back on, Facebook, or data centers, or smart phones, isn't going to move the needle toward a more caring and empathetic society. Those are all red herrings. Build schools. Pay teachers. Push back on fearmongering, xenophobia, and greed. Educating people to be good participants in the global community is the only chance we have.

I do see the failure of either party to produce a truly constructively inspiring candidate as something seriously wrong with the system. Thanks for the link, it does seem interesting.

I was more referring to "everyone else" as apathetic. Generally, I don't think voting is itself considered an act of activism. Somebody will always win the election regardless of overall levels of activism and apathy. January 6 could be considered activism, however (the bad kind that's closer to a riot). "Get out the vote" grassroots efforts are activism.

I think there is currently a general lack of enthusiasm for the political system and process in the US, and a lack of faith in the government. Trump has tapped into that, and riled people up. Tapping into fear and aggression and anger does not inspire what I would consider to be traditional political and civic activism which IMO is more rooted in "let's work together to make things better for our people."

As opposed to, "let's say MAGA but actually start foreign and domestic conflicts and squander resources on pointless personal ego projects." It is disturbing to me that people continue to support that but I'm still calling it more apathetic than activist.