Hacker News Reader

> The problem is that these refrigerator cases are not as simple as your refrigerator at home. As a simple example, many systems allow you have control over the deadband: the exact low temperature at which the compressor turns off and the high temperature at which the compressor turns back on. You also have control over defrost cycles, compressor pressure and many other things (I worked with this a long while ago, and I don't remember all the details). … This is not something that a non-expert supermarket employee would be able to handle well.

I still don't see why the expert has to be remote, or why an integrated computer can't do the optimization. Or why it needs to be fiddled with at any significant frequency.

  > I still don't see why the expert has to be remote

Because there are 1000 store locations. You're either then paying for 1000 mostly idle engineers, or somebody not really qualified has to don that hat, or you're flying your engineers around constantly. Or just connect the devices to the Wild Wild Web and be done with it.

A lot of retailers like CVS and Walgreens (~16,000 stores between them) use regional contractors, they work for numerous large companies including Walmart if needed. You don't pay for idle engineers per se. Instead you pay a relatively work rate: high cost to show-up, high hourly rate.

Oh so its optimal to maintain a security team to secure your network but not to have on site tech

I site tech for a super market?

are you going to pay even more for meat? its already expensive

You think the cost comes from expensive local tech guy on the supermarket?

Cost is influenced by all operational expenses. Think about how much more it would cost to do your laundry if the laundramat had a full time repair tech on site.

I mean it was every year before this one yes

>>just connect the devices to the Wild Wild Web and be done with it.

Yup, they're done with it, all right. "It" being all the food being cooled for sale this week. Done.

They do not need engineers at every location to turn every knob. They need to define procedures and limits, and the ability to call the engineers when things go out of those limits. Even putting only remote MONITORING online, but requiring the local person to check and turn the knobs/press the buttons. That also adds an extra set of real eyes on the situation — when the engineer calls the local dept manager to start the defrost because the hacked line says it is now time to defrost, the local manager —with eyes on the freezer— can tell him "NO!" because this was not on their schedule and the freezers are all stocked, not emptied for the defrost.

Instead, the defrost cycle just turned on in the middle of the night, and neither the now-remote engineers nor the local managers knew until it was too late. ——Again.

Security is always a trade-off against convenience. And the idiot human managers almost always all pick convenience because their managers incentivize profit, and security costs and convenience appears to save up front. They nevermind the later costs because they'll already be promoted up or job-hopped out.

You're too optimistic. I work as consultant in industrial controls. I often get called for:

- "machine doesn't work!" It was set in the wrong mode. Who did it? No answer

- "machine doesn't work!" The HMI shows a huge message with "CHECK SENSOR AA-123" or "VALVE BB-321 CLOSED" and they are puzzled when you point it to them

See also the "PLC program changed itself!" in-joke in industrial automation circles.

The reality is that what you envisage "does not scale", to borrow from software engineering.

Thank you, very informative, and having had to support biz/network software I wrote, entirely credible.

So, how can this be solved? Ideally for the long-term, but also for the current interim state — we have a known attack vector costing millions, known exposed, and need to do something.

Short term, seems like disable defrost or other critical vectors? IDK the architecture, so IDK how difficult it would be to add a local interlock or override for critical instructions that would damage the whole contents, so the local manager must also confirm "READY TO DEFROST"?

Long term, seems the only solution is to rearchitect a truly secure solution with secure access controls, locked-down to a limited set of IP and MAC addresses, all traffic encrypted, etc. Plus a deep evaluation of the sets of instructions that could be catastrophic, and adding requirement for local confirmation/override to execute those functions.

It also seems there is a lot of available improvement in the UI so regular Joe-managers could use it. Even if they see "CHECK SENSOR AA-123", their eyes glaze over because they just see it as the gateway to a hellish maze they don't want to enter. Adding layers of actually informative error messages and walk-them-thru steps to fix it seems well within our technology. It'd probably require an additional local mini-PC on each site, but could save millions.

If this gets scaled to a whole country, when most of our food supply is Just In Time, it is no longer a prank.

If they hired an extra 20 managers across 1000 stores to deal with twiddling fridge knobs over the phone, we'd be complaining about that instead. And someone would still call up managers and social engineer them to setting the knobs wrong.

They wouldn't have to hire 20 extra managers, they just would need to develop some materials and train the managers they have.

And social engineering 1000 managers is FAR more work than scaling a software hack across 1000 machines, and also far less likely to succeed. At least one of those managers is going to get suspicious and call corporate first. Moreover, it will be pretty hard to get an on-site manager looking right at the full freezers to set them to defrost....

Security and convenience are diametrically opposed and stores are specifically built around convenience. I’m not sure it’s obvious at all what the right cutoff level is.

Considering food is a national security item, the threshold must be higher.

One store loosing its frozen food is an inconvenience and insurance claim, and one electric substation failing is an inconvenience for a few days.

But whenever any one of these attacks could be scaled up, as is obviously possible with online access enabled, and the consequences become a regionally or national security threat, enabling such access must come with the seriousness of implementing national security.

Unfortunately, it is not sexy for politicians to take national security seriously, and they will get inundated with lobbying decrying the costs. Until the losses to the companies exceed the costs of security, when they come screaming to the govt for help, and demanding why they didn't do something.

It’s some frozen food, it’s not botulism in the rice. It’ll be hard to convince people of how big of a deal this truly is.

No question, it'll be hard to convince most people of how big a deal it really is.

It is the same with almost every systemic problem that seems small now but is cumulative and/or exponential.

And for sure, if it is a few stores, it is just an expensive prank. But scale it to the size of a nation or region in a country with a largely just-in-time food supply, and you're a week away from food riots, particularly if it is combined with other attacks, e.g., on the electrical and distribution system. Not a shot fired, and the country is in chaos in a week. Because no one bothered to harden the systems that seem trivial but are critical at scale.

I thought this conversation was over, but then I saw attacks had moved on to fridges storing blood in hospitals. This is an insanely important update to this conversation, should anyone happen across it.

It took no time at all for this to become deadly.

It's an obvious decision in hindsight. Not so beforehand.

This lot were already in the hindsight condition. The same thing happened all across France three weeks ago. THREE WEEKS. It was already obvious this could happen.

Of course this is not enough time to entirely change one's strategy, but it is a lot of time to install hardening measures, such as stronger passwords, limiting access to only MAC addresses of specific machines, taking access offline wherever possible, disabling the defrost cycle, etc., etc., etc., and that is just the obvious.

One might wonder how we ever managed to do it before the Internet existed.

Less efficiently. Supermarkets compete with each other on price. Margins are very thin, and supermarkets are much more efficient than they were decades ago.

>Supermarkets compete with each other on price. Margins are very thin, and supermarkets are much more efficient than they were decades ago.

That's too generic. How much exactly do these "connected fridges" contribute to that?

Saving energy (and hence electricity costs) and enabling timely maintainance.

https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=supe...

Of course that would be the pretext, but that doesn't even begin to answer the question.

How much does these savings contribute to their overall costs?

How much difference in that over dumb fridges?

How much difference in that over what can be replicated by a localized control system, at most sending reports through an "output only" line?

And what exactly would remote repair fix if some fridge hardware fails (that's not just switching to a failover hardware, that could just as well be decided by a local controller)? Since the answer is close to nothing, how much of the "remote repair" need isn't cyclical, addressing issues coming from complicating their fridges to the degree that remote repair is required?

>Of course that would be the pretext

Right. Operating efficiency is just the pretext. Congratulations on uncovering the real story about networked supermarket fridges.

In all honesty, I can't prove that these systems help supermarkets reduce their operating costs. It just seems the overwhelmingly obvious and likely explanation for their adoption. If you want to argue otherwise, then I'd say you owe us more than just a list of skeptical questions.

Dial-up modems were considered fully secure by comparison, when the remote appliance has its own private line.

That's my thought too.

These fridges may need more complex control for a variety of reasons, but that doesn't mean they have to be connected to _the internet_. To a local network for control, sure.

Put your security measures into the network itself, but I really do feel fridges are another thing which do not need to be IoT devices.

Local network, VPNs, bastion hosts, and so on. It really isn’t that hard.

Exactly this, I wonder why this setup (with ability for external techies to be granted temporary access) was not chosen in the first place. Considering the threat model, "my freezers all defrost at the same time all over my stores because of ~cyber~" should be a critical item to be discussed

> I still don't see why the expert has to be remote, or why an integrated computer can't do the optimization. Or why it needs to be fiddled with at any significant frequency.

Well, it doesn't. The retailer has a team of techs (or an outsourced vendor) rotating between stores, who are cleaning and replacing air filers, checking lubricants, and so on. They could certainly apply occasional changes to settings.

But to the product manufacturer - the remote management feature literally requires less than $2 of hardware. Hell, it might even save money, by removing screens and buttons.

And the person buying the refrigerator wants something that logs and monitors temperatures, for food safety. And they don't know anything about cybersecurity.

So when the manufacturer says that for just a little more they have a remote monitoring option, which not only provides the monitoring needed for food safety but also reduces maintenance costs - it sounds good to them. They don't realise they're signing up for a multi-million-dollar loss when they get hacked and have to throw out a load of products.

> And they don't know anything about cybersecurity

And this is the key fact. It’s no longer acceptable to sell food and not know anything about cybersecurity the same way it hasn’t been legal to drive a truck without a license in decades (because a lot of people died). There was ample warning since the incident at a couple US military bases earlier this year. Market owners need to face it.

It should be the fridge company's problem to secure their fridges. These aren't bought, they are rented as a cooling service. The supermarket owner is paying a certain amount per month to make the fridges someone else's problem, so that someone else is required to handle the problem.

If what they rent is not freezers, but refrigeration, then they have a chance of having some sort of compensation for the loss of service, unless managing the fridges and their network is the market's responsibility.

The manufacturer/operator could, at least, not come with the omnipresent admin/admin credentials combo.

> why it needs to be fiddled with at any significant frequency

Weather changes, equipment performs differently as it wears, changes in customer behavior, different items being stocked, electricity cost updates...

> I still don't see why the expert has to be remote

Because then he can get paid more.

> Even if you only save $500/store/year, you can see that it really adds up over a chain of 1000 stores.

People always say stuff like this, but it never tracks for me. If you’re a chain with 1,000 stores, $500,000 is barely a blip on your budget sheet in exactly the same way that $500 is barely a blip on the budget of a single store.

Plus there’s no way this remote management stuff doesn’t cost at least $500/store in operating costs.

> Plus there’s no way this remote management stuff doesn’t cost at least $500/store in operating costs.

Going by the number of companies that still can't grasp having to pay for ongoing maintenance for and IoT system, even with CRA and NIS2, some still selling stuff running on 12+ year old software, I wouldn't expect them to pay much for this service or to even properly maintain it.

Don't compare it to the entire balance sheet, compare it to how much you'd have to pay someone to optimize it.

At $500/store/year times 1,000 stores, you can pay an entire team of experts for a year to come up with a way to save that money, and come out ahead. That team's manager can then go to their boss and say, look, I saved the company half a million dollars a year, and they'll get a nice bonus from it.

> $500,000 is barely a blip on your budget sheet

Famous last words.

I don't think people question remote accessibility as much as they question why remote accessibility is a matter of guessing the standard password.

You can have remote access through secure means. You might even have a web page open to the public if you secure some kind of safe login mechanism.

If your story is true, then these stores and chains better take the financial impact they hit out of their CEO's paycut, because they absolutely chose to risk it all for some minor cost savings. Maybe try to sue the manufacturer (though I doubt the judge will side with you if you're hit with the same attack just a week or so after the hack appeared on the news, without having fixed the problem).

FWIW, Reuters is saying the total losses are in the range of 10's of millions of euros across all stores.

And how much money the shop looses from the attack? It sounds it wiped out all the saving remote automation brought.

Some people have to learn the hard way, I guess.

Ha! I bet nobody learned anything.

Almost everything you described is either better set at factory (or at coolant recharge - like compressor pressure, etc), or tunable on the antique mercury switch thermostat we had growing up.

The only remaining thing is the defrost cycle stuff, and even that can be 90% of the way to optimal just by having a clock that knows when peak power is. The classic thermostat we replaced the mercury switch had that. I imagine they're like $10 retail, $1 for the microcontroller.

Assuming the extra complexity does actually save another $500/store/year, how much of that $500 is eaten by maintenance contracts, software patches, wifi debugging, etc, etc? Also, what happens if the internet goes down?

I'm not convinced it makes any sense for these things to be online.

It would still be miles better to add something to do on the hardware side. For example: every time a new firmware or setting is pushed, a physical employee has to go press a button (or turn a key) to make it apply.

It would be a more sensible tradeoff bo matter the scale of the store, thab jeopardizing the entire fresh supply of a nation.

> “That's why this equipment is remotely accessible”

Complicated things don’t have to be connected to the internet. Most environment control systems in small buildings (typically with hundreds of sensors and actuators) aren’t. There is smart, deterministic, software involved but most of the time, no cloud.

How did these work then before the internet existed?

In a word, poorly.

Setting up and changing parameters would involve, at the very least, a technician walking a store employee through the process. Best case scenario the technician would dial in using a modem.

They weren't able to set up to work as efficiently.

They were more secure though.

Also, does all this remote management whatever stuff cost more or less than slightly better insulation, or an upgraded compressor?

Thank you for explaining. OK, so I get that it's not as simple as "there is NO reason to have remote access to a fridge". There are reasons to. But there are also reasons not to. I hope the decision-makers involved will start carefully thinking about the trade-off involved.

Maybe they $500/store/year is not worth the risk? Maybe having read-only remote access will still save $300/store/year and $200/store/year is not worth the risk? Maybe having a physical button that someone at the store needs to push to temporarily authorize write access is worth the hassle? etc.

This is one of those "it's all so complicated" answers that ignores the fact that for most of the past century they managed to do this without the thing that you're claiming is now necessary.

And the real answer comes out eventually: it's necessary now because we consolidated and cut costs and jobs. The people left who we pay next to nothing could never even handle them at the level we had then, because we laid off everyone who knew that old stuff.

> Even if you only save $500/store/year, you can see that it really adds up over a chain of 1000 stores.

And even if an event like this only happens once in 20 years, all of those savings evaporate. Thank god nobody will get fired for this. Or at least the first wave - it will be insane if nobody at the chain hit by this current wave gets fired.

>they managed to do this without the thing that you're claiming is now necessary.

They managed to do it when none of their competitors was doing it either. It's a low margin business.

> And even if an event like this only happens once in 20 years, all of those savings evaporate.

I don't think the numbers on that actually track, if you look at the estimated damages that are available. At once every 20 years, you'd still come out ahead.

I don’t doubt the technical details but on the economics: surely saving a small amount of money per store is still a small amount of money per thousand stores? If each store costs $500k per year in staff, maintenance, and power then the savings from having fancy refrigeration units mean I can run one more store on top of the 1,000 I already have. It doesn’t seem like much, but I am a business illiterate computer programmer not an exec!

If margins are already at the least possible bottom (according to market logic this should always happen with commodities and competition) then you need to start to look for ways to reduce your operating costs in order to increase them again

Why? If you are already at an optimum (which "least possible bottom" indicates), then any kind of you trying to move into a different state, is going to leave the optimum.

If only 50$ of those savings were invested in Tailscale.

Or netbird. I was exactly thinking the same btw clearly we're missing out something. or maybe cybersecurity is a cow ready to be milked?

Did Tailscale not have a major vulnerability just a few days ago?

For a specific service only, not the networks. But that doesn't even matter that much - if you require the attacker to get both into the network and into the device, that's a significant improvement.

$0 for Wireguard.

Only if your time is free.

We actually had freezers in stores back before the internet was widely used.

People find it more convenient to network everything, but it isn't Necessary we're seeing the very predictable downside.

I think there can be a compromise. A range of acceptable of operating values is chosen, and remote control is allowed within that range. But dangerous values like turning the freezer off or setting it to defrost should require more ceremony.

I don't see why saving $500 per store a year is better than exposing your systems to cyber attacks.

I used to talk to a supermarket chain COO (or whatever the guy who handles the operations of all stores of that brand) in China. He told me that supermarket has extremely low profit margin that sometimes a couple of unexpected bad days would turn the number of whole year off. I can imagine they would like to squeeze every cent out of it.

Grocery is a very low margin business, and they look for savings wherever they can.

"That will never happen to us" - until it does.

It’s not even “That will never happen to us” it’s “Joe, I need the WiFi password for the new fridge, do you remember what it is?” “Sure it’s $t0reNam3secur3dbec4useitsin13372023”

It was a better deal before AI, but now the economics of a lot of things are changing rapidly.

And yet... if they had instead invested in making the equipment operable locally we wouldn't be here today. This is the problem with thinking of your users as "nontechnical" lesser beings. The engineering problem becomes remove control and ownership from the user instead of make the product usable. In this case by removing control from the user they simply gave it to an attacker. Great job, guys.

I addition to the technical knowledge needed to make this work well, there are misaligned incentives between the store managers and the company as a whole.

One particular example: store managers want a very cold, very narrow temperature deadband on their frozen items. This gives the customers a nice experience. They're less likely to complain about melty ice cream. But this type of setup leads to more energy expense, and failures on the compressors due to excessive cycling.

Could this control problem be solved locally? Like is there some intuitive setting you could give an end user that still results in efficient running behavior? Wouldn't that be strictly better than putting it on the internet anyway?

>The problem is that these refrigerator cases are not as simple as your refrigerator at home

And yet they existed, and didn't need a connection to the net in the 80s and 90s. Most supermarkets not even in the 00s.