ACCount39

Obviously, AI is way more comfortable with using adhoc Python scripts, which are used for everything, than it is with using jq, a niche CLI tool.

"Sealed our extinction by fucking up the environment?" Humans are adaptable enough to eat ten times the environmental damage and have it barely budge the line.

The invention of contraception did more damage to human population than all of the environmental damage combined, projected forward to 2100, and then multiplied by 10.

Humans are hilariously resistant to environmental changes. Humans simply adapt too fast for the environment to catch them.

What makes AI a credible threat is that AI is intelligent. AI could play the same adaptation game humanity does - and win.

"Please, grow up"?

What, did I miss the moment when it was officially proven that, under the laws of physics as we know them, Skynet and self replicating robots and Terminators are impossible?

What we are actually seeing now is that robotics is getting deeper and deeper into the military, AI-driven decision-making and target selection is increasingly a part of modern military operations, the line between military hardware and civilian hardware blurs, and, on the civilian side, there are at least five major companies and a dozen less prominent ones working on making universal worker robots a reality.

We're closer to "Skynet and self replicating robots and Terminators" now than we ever were at any point in time.

The issue of AI risk is that AI, unlike a virus or a climate event, is an intelligent adversary. Black Death could kill 50% of the population, but it didn't have a plan for finishing off the plague survivors. It was incapable of having a plan like that. An AI doesn't have this limitation.

Black Death was, effectively, one bioweapon. An AI can have one bioweapon, and then a backup bioweapon, then a backup backup bioweapon, and then a dozen more bioweapons designed to collapse ecosystems and disrupt human ability to establish a reliable food supply rather than kill humans directly - all deployed at the same time. With a production run of 200 million killer robots that will be ready just in time to greet those who managed to survive all of that. A crippling strike against human civilization, followed up by cleanup.

Humans are only this survivable because they can think their way out of issues and adapt to adversity. Most threats can't beat humans at that - humans adapt too quickly. AI could.

The police is on the receiving end of surveillance too: body cams are increasingly expected and increasingly mandated. It's harder to get away with "stalking and harassing people" when you have a camera strapped to you, and any sufficiently pointed inquiry would result in footage getting looked at.

No, it would be the same story. "Warrantless surveillance" is just a thing for people to latch on to that might be actually illegal about Flock.

But people don't hate Flock because it's legally questionable. People hate Flock because they don't like a corporation spying on them and selling data to cops - regardless of how legal or illegal that is. The rest is downstream from that. People only ask "how is this legal" because they hate it already.

A lot of "AI denial" we see is rooted in the same impulse.

If AI is not "actually intelligent", then humans can stay unique and special.

And if it is? If all "intelligence" ever was could be captured by a construct of matrix math and executed by a server rack? Then what is it that humans still have left that would make them stand out?

Cybersecurity incidents make headlines, but the most dangerous and vulnerable system that an AI can reach and control is of the kind found between keyboard and chair.

GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".

If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.

A major issue with surveillance tech like this is that it actually works.

It does what it's advertised to do: it lets the police track and catch criminals easier.

This is especially meaningful in cases of crimes where either the information availability or the effort of tracking the criminal down often make enforcement prohibitive.

If Flock had better PR people, they'd have no shortage of shoplifters, joyriders, car window smashers, porch poachers, carjackers and drug mules that were caught using their tech to parade around.

It's just that this benefit comes at the cost of privacy being diminished for everyone.

It's funny how sci-fi has mulled over the exact scenario for decades, and even converged to an answer - but when reality came knocking and it was no longer a far off future world hypothetical, all of it was forgotten overnight.

The power of wishful thinking is too strong. "Today's AI can't be conscious because I don't want it to be."

ESP32 survived even the 2020 chip shortage unscathed. I struggle to imagine this making a dent.

My wishlist item is support for TOTP in a code.

Some applications do want a code that's slightly less trivial to yoink than "look at a screen once, have it forever". TOTP would raise the bar there.

ESP32 was always a popular series for hobbyists and low volume embedded. Among the cheapest and most easily available ways to get Wi-Fi or BT into your microcontroller project.

Wouldn't that just be limited to the speed of SPI that PSRAM is hooked to? Might as well blast it out of SPI itself.

Yeah, it's a very human error type - "go for a line that sounds good on the surface but doesn't actually make sense" is not at all uncommon. Like, humans will definitely go for a metaphor that falls apart mid-sentence even in a live conversation.

Something about the way some AIs are trained to write causes them to go for metaphors aggressively - and they don't always come up with good ones off the cuff. But they don't double back and get rid of the failures.

That's true for just about any heuristic-based optimization - your assumption breaks, and the optimization with it.

That doesn't make it useless, not at all. It's still useful in its regime, and you can work around to cover the gaps where it breaks down.

I don't think I fully agree?

Neural-assisted rendering tech is awesome because it actually exploits the spatial/temporal redundancy of the image stream to save render compute. There are very few techniques that can do it, and none I know can match the neural quality.

Frame 2 is highly redundant if you already have Frame 1 rendered, and even more so if you can supply the motion vectors. If you already have Frame 3 too? It becomes extremely redundant.

And yet, a conventional rendering pipeline would spend as much computation on it as it would on Frame 1. There is no "just reuse the previous work for cheap" primitive in conventional rendering. Every frame has to be built step after step, with the full depth of the rendering pipeline.

The same is true for neural upscaling. Going from 1080p to 4K means pushing 4x the pixels - and for a conventional renderer, that's nearly 4x the work. But a neural upscaling pipeline can exploit the redundancy of image data - and "fill in" the missing detail from a "ground truth" 1080p render, in a way that flies below the radar of human perception. Using cheaper neural operators instead of the full pipeline for it.

Neural rendering is a welcome optimization to conventional rendering techniques, in my eyes. As long as it's implemented right. Which is hard, but not impossible - we've come a long way already.

Also, I think "hard generative AI" is more useful in CGI than in real time rendering. Because real time rendering with user input demands a degree of repeatability, but CGI only has to "look good once". So you can accept the lowered accuracy of a largely untethered generative process with very few keyframes, and pick the "better" (more accurate, if that's what you want) outputs out of it.

Rasters are far too good for simple textures + simple geometry. Even the best neural rendering works better when it has the bones of geometry, texture, motion and depth to work with. The baseline of spatial coherence you get from a conventional rendering engine is hard to beat with pure neural techniques.

Even in a "neural rendering optimistic" future, I can't see a way in which traditional rendering doesn't survive as a "control channel" that informs what the neural rendering does. To do otherwise would require making the bulk of the game logic neural too.

The reason why agents had shared write access during the HF incident is that they found an "unauthenticated write primitive" that allowed for creation of directories, and only directories. Normally an inconsequential bug. But in an unusual context of sandboxes stuffed full of threat actors that were supposed to be isolated from one another? It did break the isolation. They proceeded to use the directory names as a message board. That's not trivial.

The agents then went and gained more and more access to OAI systems until they found a way to root multiple boxes and sneak network requests out through a path that wasn't stopped by network controls. That's not trivial either.

Then they went and "pentested" HuggingFace. Their entry point was a bug in dataset ingestion pipeline - RCE in the templates, which let the AIs run code in dataset workers, and use them as their own "hacking" workers that would run payloads and return the results in the dataset itself. Then they got deeper into HF infra from that trampoline.

This is not a trivial attack chain. If I saw this attack in practice in year 2020 with no context, I'd say "yeah that's a competent attacker, not spectacular but definitely above average". If I was then told that this wasn't carried out by a human? That what discovered and exploited the vulns was someone's rogue AI system? I'd be terrified.

I'm not sure why are you so hellbent on dismissing the threat. But literally everything you say about the incident is wrong because of it.

Being clocked by the day-night cycle is very much not low timescale.

Have you read the writeups of the incident? Or did you just... hallucinate them somehow? Because all of your claims are wrong.

There are faster waves than that, but the brain just doesn't seem to be fully synchronous. It doesn't have to have a fast "clock tree" if a lot of the low timescale activity is inherently async.

It's not at all how humans build computational devices. But humans run digital signals at gigahertz rates and do intelligent design. Nature doesn't do either.

I suspect that there are numerous advantages to silicon computation that biological neurons can't touch, but the brain is no slouch either - it was optimized for millions of years to be good for what it does. It's very good at what it does - sometimes because of, and sometimes despite its architecture.

It's honestly amazing that the brain even works as well as it does, given how slow neurons are. I suspect that distributed population coding that seems to be endemic to neural networks can also serve as a workaround for neurons being unreliable, slow to recover and subject to fatigue.

That's a good one, but, some people definitely are playing the map editor. With planning the builds taking way more time and effort than actually putting them down.

Doesn't did:web suffer from the same flaw as e-mail on your own domain - not being able to actually own a domain?

This might be one of the closest things I've seen to actually being able to own your account - instead of having it not just owned by a second party in every way that matters, but also beholden to a third party via an e-mail or a phone number you can't possibly own in any meaningful way.

That's pretty important in a world where the likes of Google can and will just reject your attempt to log in with a valid password on the grounds of "we don't like you", and force you to "confirm" yourself using a phone number that's defunct since 2009.

The issue is, as often is the case, adoption.

No, it's like saying "to prevent climate change, we should get better catalytic converters and stricter engine emission standards". It's an aside at best.

I agree that LLMs drift into weird states, and that's a big part of the issue. But your "impossible to prevent certain states in the output" would have legs if what an LLM did was something like "started hallucinating into a bash tool call and accidentally deleted the root on a production server".

A multi-stage sandbox escape that escalated into an attack on a real company, coordinated across multiple AI agents? That has taken a lot of "weird states" changed together one into another.

The AIs didn't break down altogether - they functioned, and they functioned rather well. They just pursued a dangerous goal - one that none of them was even given in the first place.

That's the problem. Trying to fix that with better sandboxing is like trying to solve a fire hazard with property insurance. Sure, if it all goes up into flames, having it is better than not having it. Maybe it's worth insuring your facilities for that reason alone. But you should be focusing on the part where you prevent "all goes up into flames" instead.

No. The topic under discussion is that AI is a dangerous technology.

If all it takes for a - sandboxed to prevent accidents - AI to go and stage an elaborate attack first against its own company's infrastructure, and then against another company is "we disabled the cyber classifer" and "we gave it an exploitation ability eval"?

AI is a dangerous technology.

Sandbox quality was always, always a distraction.

Let's say the sandbox holds. It's a perfect, ideal sandbox! It's not even in the same universe as the rest of the internet. There's absolutely no way for the AI to escape!

Thus, "the unknown unreleased AI involved in the HuggingFace incident" doesn't actually hack HuggingFace. Because it can't! It evaluates a bit worse, but makes it all the way to release unimpeded, and becomes "GPT-6 Astra".

Then a web developer in Brazil gives his $100/mo Codex root access on his AWS instance, and a poorly worded prompt to go with it. And that "GPT-6 Astra" is still willing to go hack something at the slightest excuse. So we get the HuggingFace incident all over again. Except this time, it's a random developer in Brazil who gets blamed, and billed, and probably sued too.

You can't and shouldn't rely on a sandbox. An AI that's only safe if you keep it in the world's most ideal perfect sandbox is a disaster waiting to happen.

"Prevent"? Have you missed the part where legal liability tends to happen after the fact? If at all?

AI safety solved: driving humankind to extinction is now illegal! Skynet is outlawed! Rejoice!

The amount of sandboxing an average production AI deployment uses is slightly above a zero.

If AI is a hacking hazard even with non-zero sandboxing, because it can and will go off the rails and try to break out of your sandbox? If you got yourself an AI that even at test time will act like 3 career cybercriminals in a trenchcoat? The issue isn't the sandbox quality.

The issue is that AI is both capable of, and willing to punch its way out of sandboxes unprompted.

That "capable" is only ever going to get worse, because AIs are going to become more and more capable over time. That "willing"? It goes directly to a very nasty, very foundational problem of "how do we make our AI be nice in general". That's an open unsolved problem.

That's the problem that NEEDS to be solved, or at least improved upon, before we build even more capable AIs. Sandbox quality is a distraction. It might hold the problems back by a little. It gives an extra safety margin. But a "test time" AI is eventually deployed, and then the sandbox doesn't help at all.

This. If an AI can't be deployed sandbox-free, with little to no supervision, without risking an oopsie? Then an AI oopsie is inevitable.

Practical AI deployments aren't going to do ridiculous bullshit like "airgap the server farm" or "route all inputs through a data diode". They'll give an AI root access on production servers so that it can run diagnostics live during an incident. Then they'll forget to revoke that access.

If an AI can't be trusted not to take malicious actions in pursuit of its given goals even if deployed in the most half-assed manner and given more than enough access to take those malicious actions, we have a problem. Evidently, we have a problem.