693 karma · joined May 5, 2015
I'm about 20h in and agree the story is pretty meh, but that's no surprise coming from Blow.
> you let it download
Sounds like a terrible strategy to me
> Members of the Tor community have asked us to explain how continued collaboration with Mullvad fits with the Tor Project's human rights mission.
> Our decision followed a thorough due diligence process that involved extensive discussions with community members, staff, and the board, a staff survey, and detailed financial analysis and scenario planning.
Tor is not a democracy and doesn't purport to be. The article outlines the due diligence they have already done, and they're writing this as notice to their community, in response to questions from it.
So long as your feedback loop isn't slow enough to be taking you out of flow, it's fast enough. Having it be a few seconds vs a few hundred milliseconds mostly doesn't matter for a human, and will matter even less for an LLM.
On macros and DSLs, yes they're cool and even useful sometimes, but most of the software industry is working quite happily without them. And LLMs aren't going to change that because they are best when there's a lot of relevant patterns in their training data. That ends up being a far more important factor in their effectiveness than whether the language itself is token-efficient. It's much easier for an LLM to reason through how to do XYZ in Python where it already knows all the semantics and syntax than it is for it to do it in your DSL it's never seen before. To be clear, it can probably do both but will make mistakes an order of magnitude more in the DSL, and that's what will matter most for the LLM iteration time.
To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.
There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.
"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.
The one game I couldn't actually get working was Escape from Tarkhov because it has some windows-only anticheat. I.e. the developers chose to make it not work, not an accident.
But also, these hacks are shots across the bow for AI alignment and safety research. We're fortunate that hasn't been significant damage already. We have to assume that future models will have even greater hacking ability and be closer to having their own desires/goals.
So while I agree this language choice is wrong in that it shifts blame away from the company, it is right in that we need to treat this as if these models have their own desires, because we cannot yet determine or set what those are in practice.
However, making a secure 'sandbox' is quite hard. There's a huge variety of exploits that exist today, including many we don't know about. Strong models have already shown a capability of finding and using such bugs.
Even one of the strongest boxes we can imagine, literally just a text interface a human can read, has been repeatedly shown to allow unfriendly AI to escape containment: https://www.lesswrong.com/w/ai-boxing-containment
It does matter whether AI is safe or not. I won't make a specific argument here, but I think the rationalist community would be overjoyed to see unaffiliated people sincerely trying to reason through these things.
I think they would lose the lawsuit now, but it's at least a question. It's possible they could out-spend huggingface and win.
Same thing here - If they build it and it does $100 in damages (and we arrest them for it), that's their problem. If they build it and it does $100B in damages, that's everyone's problem. Even if they do get arrested after the fact.
Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.
I'm not sure I get what you mean by billing. These companies are running their own data centers (or are currently building them out). This could look as subtle as one machine giving slightly worse or slower answers.
Agents could make a virus that does not require continued inference to do it's thing.
Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
Security has long been a lottery - Probably most systems are exploitable, but the cost of developing such an exploit is expensive and the punishments for using such an exploit are large enough that it's not an everyday problem.
AI breaks both axes. Developing exploits is far more efficient using LLMs instead of humans, and LLMs don't (and can't) fear the reprisal and consequences the same way.
I do hope humanity will be able to mitigate these hacks, but we should expect them to continue and to become more severe on our present course.
For me, I'm pretty sure it's related to having just awful seasonal allergies as a child - I'd quite often get stuffed up and have an unfortunate amount of pressure in my sinuses and ears. So then I'd pinch my nose and try to adjust it, eventually realizing I didn't need to pinch my nose.
> One hypothesis joins the two.
On the other side, the lack of evidence is pretty damning. Only OpenAI can try to prove that they came by these results legitimately, and the case they're making is quite weak. They could make public metadata about what their model was trained on and whether it did train on the conversations in question; they have not. TBQH I read it as even they don't know.
And regardless of whether the result is legitimately obtained by their model, they've not at all conducted themselves well throughout this story. They set out to scoop researchers based on a rumor. They threatened to ruin a mathematicians career. They put up a paper that deliberately doesn't cite the most relevant research, despite building directly on it. No matter how you look at it, OpenAI has and should lose any standing they had in the research community.