Hacker News Reader

Comment by zahlman | Hacker News Reader

> The problem is that these refrigerator cases are not as simple as your refrigerator at home. As a simple example, many systems allow you have control over the deadband: the exact low temperature at which the compressor turns off and the high temperature at which the compressor turns back on. You also have control over defrost cycles, compressor pressure and many other things (I worked with this a long while ago, and I don't remember all the details). … This is not something that a non-expert supermarket employee would be able to handle well.

I still don't see why the expert has to be remote, or why an integrated computer can't do the optimization. Or why it needs to be fiddled with at any significant frequency.

View on HN

  > I still don't see why the expert has to be remote

Because there are 1000 store locations. You're either then paying for 1000 mostly idle engineers, or somebody not really qualified has to don that hat, or you're flying your engineers around constantly. Or just connect the devices to the Wild Wild Web and be done with it.

>>just connect the devices to the Wild Wild Web and be done with it.

Yup, they're done with it, all right. "It" being all the food being cooled for sale this week. Done.

They do not need engineers at every location to turn every knob. They need to define procedures and limits, and the ability to call the engineers when things go out of those limits. Even putting only remote MONITORING online, but requiring the local person to check and turn the knobs/press the buttons. That also adds an extra set of real eyes on the situation — when the engineer calls the local dept manager to start the defrost because the hacked line says it is now time to defrost, the local manager —with eyes on the freezer— can tell him "NO!" because this was not on their schedule and the freezers are all stocked, not emptied for the defrost.

Instead, the defrost cycle just turned on in the middle of the night, and neither the now-remote engineers nor the local managers knew until it was too late. ——Again.

Security is always a trade-off against convenience. And the idiot human managers almost always all pick convenience because their managers incentivize profit, and security costs and convenience appears to save up front. They nevermind the later costs because they'll already be promoted up or job-hopped out.

It's an obvious decision in hindsight. Not so beforehand.

This lot were already in the hindsight condition. The same thing happened all across France three weeks ago. THREE WEEKS. It was already obvious this could happen.

Of course this is not enough time to entirely change one's strategy, but it is a lot of time to install hardening measures, such as stronger passwords, limiting access to only MAC addresses of specific machines, taking access offline wherever possible, disabling the defrost cycle, etc., etc., etc., and that is just the obvious.

Security and convenience are diametrically opposed and stores are specifically built around convenience. I’m not sure it’s obvious at all what the right cutoff level is.

Considering food is a national security item, the threshold must be higher.

One store loosing its frozen food is an inconvenience and insurance claim, and one electric substation failing is an inconvenience for a few days.

But whenever any one of these attacks could be scaled up, as is obviously possible with online access enabled, and the consequences become a regionally or national security threat, enabling such access must come with the seriousness of implementing national security.

Unfortunately, it is not sexy for politicians to take national security seriously, and they will get inundated with lobbying decrying the costs. Until the losses to the companies exceed the costs of security, when they come screaming to the govt for help, and demanding why they didn't do something.

It’s some frozen food, it’s not botulism in the rice. It’ll be hard to convince people of how big of a deal this truly is.

No question, it'll be hard to convince most people of how big a deal it really is.

It is the same with almost every systemic problem that seems small now but is cumulative and/or exponential.

And for sure, if it is a few stores, it is just an expensive prank. But scale it to the size of a nation or region in a country with a largely just-in-time food supply, and you're a week away from food riots, particularly if it is combined with other attacks, e.g., on the electrical and distribution system. Not a shot fired, and the country is in chaos in a week. Because no one bothered to harden the systems that seem trivial but are critical at scale.

I thought this conversation was over, but then I saw attacks had moved on to fridges storing blood in hospitals. This is an insanely important update to this conversation, should anyone happen across it.

It took no time at all for this to become deadly.

If they hired an extra 20 managers across 1000 stores to deal with twiddling fridge knobs over the phone, we'd be complaining about that instead. And someone would still call up managers and social engineer them to setting the knobs wrong.

They wouldn't have to hire 20 extra managers, they just would need to develop some materials and train the managers they have.

And social engineering 1000 managers is FAR more work than scaling a software hack across 1000 machines, and also far less likely to succeed. At least one of those managers is going to get suspicious and call corporate first. Moreover, it will be pretty hard to get an on-site manager looking right at the full freezers to set them to defrost....

You're too optimistic. I work as consultant in industrial controls. I often get called for:

- "machine doesn't work!" It was set in the wrong mode. Who did it? No answer

- "machine doesn't work!" The HMI shows a huge message with "CHECK SENSOR AA-123" or "VALVE BB-321 CLOSED" and they are puzzled when you point it to them

See also the "PLC program changed itself!" in-joke in industrial automation circles.

The reality is that what you envisage "does not scale", to borrow from software engineering.

Thank you, very informative, and having had to support biz/network software I wrote, entirely credible.

So, how can this be solved? Ideally for the long-term, but also for the current interim state — we have a known attack vector costing millions, known exposed, and need to do something.

Short term, seems like disable defrost or other critical vectors? IDK the architecture, so IDK how difficult it would be to add a local interlock or override for critical instructions that would damage the whole contents, so the local manager must also confirm "READY TO DEFROST"?

Long term, seems the only solution is to rearchitect a truly secure solution with secure access controls, locked-down to a limited set of IP and MAC addresses, all traffic encrypted, etc. Plus a deep evaluation of the sets of instructions that could be catastrophic, and adding requirement for local confirmation/override to execute those functions.

It also seems there is a lot of available improvement in the UI so regular Joe-managers could use it. Even if they see "CHECK SENSOR AA-123", their eyes glaze over because they just see it as the gateway to a hellish maze they don't want to enter. Adding layers of actually informative error messages and walk-them-thru steps to fix it seems well within our technology. It'd probably require an additional local mini-PC on each site, but could save millions.

If this gets scaled to a whole country, when most of our food supply is Just In Time, it is no longer a prank.

A lot of retailers like CVS and Walgreens (~16,000 stores between them) use regional contractors, they work for numerous large companies including Walmart if needed. You don't pay for idle engineers per se. Instead you pay a relatively work rate: high cost to show-up, high hourly rate.

Oh so its optimal to maintain a security team to secure your network but not to have on site tech

I mean it was every year before this one yes

are you going to pay even more for meat? its already expensive

You think the cost comes from expensive local tech guy on the supermarket?

Cost is influenced by all operational expenses. Think about how much more it would cost to do your laundry if the laundramat had a full time repair tech on site.

I site tech for a super market?

One might wonder how we ever managed to do it before the Internet existed.

Less efficiently. Supermarkets compete with each other on price. Margins are very thin, and supermarkets are much more efficient than they were decades ago.

>Supermarkets compete with each other on price. Margins are very thin, and supermarkets are much more efficient than they were decades ago.

That's too generic. How much exactly do these "connected fridges" contribute to that?

Saving energy (and hence electricity costs) and enabling timely maintainance.

https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=supe...

Of course that would be the pretext, but that doesn't even begin to answer the question.

How much does these savings contribute to their overall costs?

How much difference in that over dumb fridges?

How much difference in that over what can be replicated by a localized control system, at most sending reports through an "output only" line?

And what exactly would remote repair fix if some fridge hardware fails (that's not just switching to a failover hardware, that could just as well be decided by a local controller)? Since the answer is close to nothing, how much of the "remote repair" need isn't cyclical, addressing issues coming from complicating their fridges to the degree that remote repair is required?

>Of course that would be the pretext

Right. Operating efficiency is just the pretext. Congratulations on uncovering the real story about networked supermarket fridges.

In all honesty, I can't prove that these systems help supermarkets reduce their operating costs. It just seems the overwhelmingly obvious and likely explanation for their adoption. If you want to argue otherwise, then I'd say you owe us more than just a list of skeptical questions.

Dial-up modems were considered fully secure by comparison, when the remote appliance has its own private line.

That's my thought too.

These fridges may need more complex control for a variety of reasons, but that doesn't mean they have to be connected to _the internet_. To a local network for control, sure.

Put your security measures into the network itself, but I really do feel fridges are another thing which do not need to be IoT devices.

Local network, VPNs, bastion hosts, and so on. It really isn’t that hard.

Exactly this, I wonder why this setup (with ability for external techies to be granted temporary access) was not chosen in the first place. Considering the threat model, "my freezers all defrost at the same time all over my stores because of ~cyber~" should be a critical item to be discussed

> I still don't see why the expert has to be remote, or why an integrated computer can't do the optimization. Or why it needs to be fiddled with at any significant frequency.

Well, it doesn't. The retailer has a team of techs (or an outsourced vendor) rotating between stores, who are cleaning and replacing air filers, checking lubricants, and so on. They could certainly apply occasional changes to settings.

But to the product manufacturer - the remote management feature literally requires less than $2 of hardware. Hell, it might even save money, by removing screens and buttons.

And the person buying the refrigerator wants something that logs and monitors temperatures, for food safety. And they don't know anything about cybersecurity.

So when the manufacturer says that for just a little more they have a remote monitoring option, which not only provides the monitoring needed for food safety but also reduces maintenance costs - it sounds good to them. They don't realise they're signing up for a multi-million-dollar loss when they get hacked and have to throw out a load of products.

> And they don't know anything about cybersecurity

And this is the key fact. It’s no longer acceptable to sell food and not know anything about cybersecurity the same way it hasn’t been legal to drive a truck without a license in decades (because a lot of people died). There was ample warning since the incident at a couple US military bases earlier this year. Market owners need to face it.

It should be the fridge company's problem to secure their fridges. These aren't bought, they are rented as a cooling service. The supermarket owner is paying a certain amount per month to make the fridges someone else's problem, so that someone else is required to handle the problem.

If what they rent is not freezers, but refrigeration, then they have a chance of having some sort of compensation for the loss of service, unless managing the fridges and their network is the market's responsibility.

The manufacturer/operator could, at least, not come with the omnipresent admin/admin credentials combo.

> why it needs to be fiddled with at any significant frequency

Weather changes, equipment performs differently as it wears, changes in customer behavior, different items being stocked, electricity cost updates...

> I still don't see why the expert has to be remote

Because then he can get paid more.

Reply on news.ycombinator.com