drillsteps5

To be fair, their entire threat model was likely "who TF would want to break in to change the settings on a thermostat on a fridge at a grocery store".

A valid question which has yet to be answered (at least not publicly).

On a second thought, this might not be classified as theft as there was no intent to steal. Software malfunctioned and as a result cryptocurrency was used to pay for whatever service they it was used to pay for. So as long as it's reimbursed they might be fine.

Although at some point it will start bordering on negligence, running software known to do all kind of weird and unexpected things. With each disclosure they're getting closer and closer to that point...

>Making an unauthorized purchase with someone else's cryptocurrency

I know we love inventing new words but "stealing" is much more descriptive

>...It [the model] then reasoned with itself that since it found the information publicly, it could use it... it figured that since the wallet it found was public, that "funds could be stolen anytime".

Can someone maybe provide more details as to what actually happened? "the wallet was public"?..

Update: from OpenAI's own report it seems the models found exposed private keys of some Eth developers along with recovery passphrases and thus were able to get access to wallets. Does it qualify as "public wallet"?..

I think this comes down to the implementation of a particular system. But in general, since control system (the one that controls things through actuators) is NOT online, one can't interfere with it through online channels. They can interfere with sensors and fake readings reported online for example, probably, but not the ones used by the control system as it's offline.

There're different devices. You put online some but not the others.

Temperature sensor is a sensor. Refrigeration compressor motor is an actuator.

GPS positioning of a car is a sensor. Motor to turn the wheel is an actuator.

Aircraft altimeter is a sensor. Motors to control control surfaces are actuators.

And so on.

Pretty much.

Some devices are sensors, like temperature sensors. It's not that they're "read only", it's that... You can't override temperature. For that, you need other devices. Like a switch to a motor that operates refrigeration compressor. That would be an actuator. If you put the data from the sensor online, the worst case scenario is someone unauthorized gets your data. If you build interface to control an actuator online... You can read the article up top.

The problem is that all major cloud providers (AWS and Azure for sure, and I'm sure Google as well) have a large number of services targeting IoT devices, both to read and control. So default architectural decision is "We'll just plug these into AWS IoT, it's secure, reliable, highly available, and has all the features we need, all we have to do is to configure what we want and we're all set". Which in reality is not that secure just because of how IoT is designed. But AWS/Azure obviously push for these solutions because it gives them more revenue.

Alternative theory: Google's "mea culpa" on "misalignment incident" while testing the agents using their "next gen frontier model" is coming out in three... two... one...

Or, as one old and wise sysadmin would often say in the early days of the Internets, "the only 100% reliable firewall is a pair of scissors"

There's huge difference between _sensors_ and _actuators_. You can put sensors online if you don't care too much about data leaking out. You put _actuators_ online... You need to be able to deal with the consequences.

I only use Wikipedia occasionally but this is seriously the worst article I've ever seen. It switches the definition between "running on my own hardware" and "running it in the cloud" multiple times. It's been pointed out in a discussion about the article several months ago but nobody bothered to correct it.

"Hosted" is the term that describes an application running on someone else's hardware. As opposed to "self-hosted" which describes an application running on my own hardware. There's no need to invent new terms, just use the existing ones properly and this will prevent any confusion.

This makes sense. You can self-host on-prem or host in the cloud. Easy and intuitive.

With the popularity of OpenClaw I am honestly shocked we haven't heard of many more incidents. I've observed people install it, give access to their Google account and everything that Google has (which includes whatever bank accounts and credit cards registered there) and tell it go do fairly complex tasks, like book a vacation at the best price. Granted, it was almost a year ago and models learned a lot since then, but I still think there's a lot of things happened that people are not aware of.

>highly goal-oriented autonomous agents

That actually made me LOL

These companies are building software. That doesn't work very well. The output it produces does make sense at times, but there are times when it doesn't. And instead of fixing that, or admitting it can't fixed, they started bolting actuators to them, executing actions online (for now) based on the output of their buggy software.

And when this results in actuators executing some bad actions they scream in horror "AI went rogue! It escaped the containment!!! It's going to kill us all!!!"

Go fix your software before you let it do stuff online or IRL. It's not "Terminator", it's just bad QC.

Several months ago, having exhausted all other options in fruitless job search, I asked for assistance of a recruiter who I used to work with some time ago. Just to get some pointers on how I could improve my search, with his experience working on the other side. His advice on "tailoring" was very simple.

Go to the most popular LLM (ie Claude). Give it the job description. Give it your resume. Ask it for percentage match, it will give you a number. Then ask for advice on improving your resume, you can either ask to rework the resume by itself (if you're in a hurry) or you can work with the thing to modify the sections/entries yourself, one by one, if you have time. With each modification you will get better and better percentage match. Ideally you need to get to high 80% or maybe even 90%.

The idea is that automated ranking tools used by recruiters/HR are ultimately use the very same LLM, so you are improving your ranking assigned to your resume by those ranking systems.

I have to add that now this advice is mostly useless as everybody's using the same technique so you won't be standing up against others but merely be on the same level.

Add to it the fact that many job descriptions are inaccurate and sometimes downright misleading, and hiring managers use their own criteria, and you will understand why connections/referrals is probably the only way to get hired now.

If you go through "Show HN" posts you will see multiple projects that partially or completely automate both search and application process. Just set up a bot that searches for newly posted positions matching whatever criteria, customizes the resume, applies. Probably costs a few cents for the applicant, but increases their chances somewhat. And the rest of us deal with the consequences.

> Such as making excessively reductive analogies that completely fail to grasp that just as "brain" is not helpfully described as "just chemistry" > despite being made of just chemistry, so too are machine learning systems not helpfully described as "just computer programs" despite being made of just > computer programs.

How computer program arrives at the result is utterly irrelevant, through explicitly written instructions or through running inference on pre-trained neural network. What matters is that it does not have agency. Its creators and operators do. So whatever the software does they are responsible for, both good (summarizing my emails for the week) and bad (gaining unauthorized access and destroying data). There's no need for new anything, it's all covered in existing legal frameworks (including presence or absence or intent).

> The bit where, even without anyone bringing up "p(doom)", a system which has the means to hack arbitrary other machines, and which appears to be > motivated to do so by accidental mis-phrasing of prompts, can obviously cause damages exceeding the USA's GDP, let alone whatever public liability > insurance the company happens to have.

Yes, absolutely, which means that building actuators that convert the output from probability-based, black box, non-deterministic systems, that are known to produce unexpected output, into actions in the real world is absolutely horrendous idea. Bizarre even.

What's your point?

OpenAI's "artificial intelligence" is an inference program that they developed which receives input and generates output. Based on which other programs, also developed and maintained by OpenAI, perform actions. Such as sending POST/GET requests to various sites which result in gaining unauthorized access and even destruction of information (deleting logs/message history) at the said sites.

What exactly requires "new regulatory framework" here? You running your software resulted in illegal actions, you are to be held liable within existing laws and regulations.

"We built a program and this program performed destructive actions. We need regulatory framework"

Make that make sense?

It wasn't bad weather. Captain was going way too fast to begin with, and started descent later than needed. So he needed to catch up with the glide/descent slope by making the plane descend faster. Thus gaining even more speed. He still arrived too late (overshot the beginning of the runway), with higher speed (I think 30 kts over the limit) and not configured (flaps not down).

It was really, really, really sloppy approach. And all could have been corrected just by executing go-around literally at any point in time up to 10 sec before he did. Both pilots still would have their careers, and those 5 people would still be alive.

Yes I was wrong. Looking at the transcript that go-around call out was likely from the PF; that's where the data recorder indicates the brakes were released the throttle set to max for the go-around (and go back to braking only 4 sec later).

Still, the PM notified PF that they were going too fast many times during the approach. All warnings were ignored.

The prevailing opinion on PPRuNe is that the approach was totally botched from many miles out and the touchdown should never have been attempted (many participants there are professional pilots). There might have been a gust of wing as they were about to touch down, but they were also way too fast and lowered the flaps very late, literally as they crossed the runway threshold, causing extra float due to ground effect.

The pilot flying appears to have been dead set on landing on the first try no matter the cost and the first officer did not intervene as he should have.

They got incredibly lucky and both survived, however likely will never be behind the controls ever again.

A few observations to add context (partially stolen from PPRuNe):

- Pilot Flying (PF) was the Captain, 55 yo (presumably) more experienced pilot, who only got certified to fly that type (B767) in May

- Pilot Monitoring (PM) was a 38 yo first officer, also not very experienced on B767

- PF made a number of mistakes, going too high, too fast (partially because he was descending too fast), not catching the glidescope on time

- PM was concerned that the approach/descent was abnormal but kept his mouth shut as the more experienced captain was at the controls (Edit: apparently he did speak up but PF did not acknowledge)

- PM lowered the flaps much later than required (PF orders flaps lowered and PM executes, but flaps can only be lowered when aircraft goes below certain speed) and the flaps were lowered all the way as the plane crossed the runway threshold, which likely caused plane to "float" and overshoot even further

- Plane touched down too late, and left wheel even later, which likely precluded some braking mechanisms from activating (or the PF forgot to since was busy trying to put the thing down)

- PM finally said "too fast, go around" but PF ignored him (or likely did not hear)

- In these situations PM has the right (and obligation) to take over the controls and start go around by themselves, but he didn't, presumably because the Cap was driving

- PF finally panicked and started the go around procedure, bit realized it was too late almost as the front wheel exited the runway, idled the engines and reapplied the brakes.

So, yeah...

Very good point, it's about exploiting inference engine itself, and not the agentic stuff.

I found it lacking details. All these things do is split the input into tokens, encode them, feed to the input, run the inference, collect the output, decode, convert to tokens, assemble the final text, and output (leaving aside multi-modal capabilities for now). What exactly can be exploited here? Aside from usual things like buffer overruns etc.

It does mention that vLLM at some point ran eval() on final output (?) which was confusing, why would it do that? It's an inference engine, not an agent.

So interesting topic, but lacks details.

Edit: many (all?) of them have http endpoints, so that obviously can be exploited, but I don't think it would qualify as exploiting inference, it's just hacking the http service.

Can you help me find a few names? These are inference providers running open weights models, right?

Thank you in advance!

>While physics is about finding the laws that govern nature, engineering (say, mechanical or electrical engineering) is about accepting these laws as foundation, and using these laws to build or improve things. This is not less scientific than physics; (classic) engineers are just experts in a special, more applied sub-discipline of physics.

>Similarly, software engineering is about applying the foundations that mathematics, electrical engineering or computer science brought in a similar scientific way.

Exactly my point, that's why most of IT professionals should be called "software engineers" and not "computer scientists", just like electrical engineering is not "electrical science" or "physics" even though they it's obviously applied physics.

A graduate with physics degree will not work as electrical engineer, what's with all the "computer science" degrees required to get into software engineering?

>It's just a label on a box to put people in.

What we call things is important because that's how we convey meaning/definition of things. Because when we say "computer scientists" people will start comparing them to other scientists which is completely inaccurate.

>Engineers don't traditionally even "build" anything and there are both upper-case and lower-case engineers in title and legal designation.

Not sure what the difference is between Engineer and engineer (again, definitions are important), but I would be ok with equating programmers with other trades...people, like electricians, car mechanics, carpenters, etc. And then higher-end IT professionals (infra/data/application architects) would be akin to engineers, designing, but not building (for the most part).

But neither category would still be scientists.

Most "computer scientists" are not scientists. They are engineers. They engineer (build) something abstract, software, but that doesn't make them scientists.

"Computer scientists" do exist, they create theoretical underpinnings of the hardware and software that millions of engineers then use to create things, but majority of people working in IT are engineers.

Now, don't get me started on "data scientists"...

To be fair, the applicants for positions on the "AGI Safety and Alignment Team, which seeks to mitigate the risks of advanced AI" will likely have extremely non-traditional background and therefore likely screened out by the LLM.

Still, this would have been hilarious if it weren't so sad...