Comment by chromaton
4 hours ago
I worked in the field of control systems for supermarkets back in 2008.
These systems are internet connected and remotely controllable because the managers need to be able to see shifts in usage over time and take action to save electricity and provide better food storage. One example might be that due to shifting customer patterns and products, you might want a certain refrigerator case to be kept colder with a narrower temperature range. Another example is for Demand Response: when a brownout may occur, the utility companies want people to lower their load, which, in the case of supermarkets, means turning off lights, increasing the thermostat on the AC, turning off dehumidifiers, etc.
One notable story from that time: this was during the 2008 economic crash, and the company went from hiring to laying off within a week. A group of technicians were working on a refrigerator compressor system and were told by security to stop what they were doing and immediately exit the building. They tried to explain that they were running a special test and needed to shut things down before leaving, but the security guys weren't having it and made them leave right away. The test ended up running overnight, and the managers came in the next morning to over $100,000 worth of damage to the refrigeration equipment.
There's huge difference between _sensors_ and _actuators_. You can put sensors online if you don't care too much about data leaking out. You put _actuators_ online... You need to be able to deal with the consequences.
How does this actually work, in practice? If you have read access to sensors, then there's a non-air-gapped network between the control systems and the internet.
If your network isn't air gapped, then it's vulnerable, and attackers will side step the "read only" path to affect control system changes.
It's being called a data diode (diodes usually allows current in one direction only).
It can be implemented with a fiber optics cable with only the transmitter on the secure side, a laser+photo cell for real air gap, cutting the rx cable in a rs232 or having a microcontroller with an output pin connected to another controllers input pin. By handling it with hardware instead of software you get a very secure partition between the trusted and untrusted side.
It's used to monitor for example industrial systems and power plants.
Wikipedia calls it unidirectional network: https://en.wikipedia.org/wiki/Unidirectional_network
3 replies →
System a emits telemetry. System b reads telemetry and exposes to the internet.
You can't drive my car simply by pointing a webcam on the steering wheel.
3 replies →
Difference between a smart thermostat for your ac and an Arduino temperature sensor next to your AC thermostat?
There're different devices. You put online some but not the others.
Temperature sensor is a sensor. Refrigeration compressor motor is an actuator.
GPS positioning of a car is a sensor. Motor to turn the wheel is an actuator.
Aircraft altimeter is a sensor. Motors to control control surfaces are actuators.
And so on.
3 replies →
Exactly, this is bad security plain and simple
This is the way. Read only remote, write access requires on site access.
Pretty much.
Some devices are sensors, like temperature sensors. It's not that they're "read only", it's that... You can't override temperature. For that, you need other devices. Like a switch to a motor that operates refrigeration compressor. That would be an actuator. If you put the data from the sensor online, the worst case scenario is someone unauthorized gets your data. If you build interface to control an actuator online... You can read the article up top.
The problem is that all major cloud providers (AWS and Azure for sure, and I'm sure Google as well) have a large number of services targeting IoT devices, both to read and control. So default architectural decision is "We'll just plug these into AWS IoT, it's secure, reliable, highly available, and has all the features we need, all we have to do is to configure what we want and we're all set". Which in reality is not that secure just because of how IoT is designed. But AWS/Azure obviously push for these solutions because it gives them more revenue.
Managers can just call the store on-site employees. Or come in person. They just want to save on labor (aka replace humans with robots).
Supermarkets very efficiently drive down the cost of food so that food is abundant - basically everyone in developed counties has access to low cost food. Part of that process is not wasting overhead on having people manually checking various gauges.
It’s a very low margin business if you don’t work efficiently you’ll get outcompeted by people who do like Aldi/ Trader Joe’s etc
Comparing Control automation of boring systems running all the time to robots is really funny to me. Those systems weren't previously run by humans before the more electronic approach, the control mechanism was physical in nature but less adjustable.
Most of the HVAC based control system changes are about saving energy, not labor.
Grocery is very low-margin, and if a new system can save $500 on electricity and maintenance per year per store, and you've got 1000 stores, the technology justifies its cost very quickly.
Saving on labor is the purpose of most technologies since the pointy stick.
Those are good reasons to have the machines network connected. They are not good reasons to have the machine internet connected.
Why not put them behind a VPN
Often attacks like this involve first getting on the VPN (leaked creds, a n-day on a web-accessible endpoint, etc) then running wild on the internal services that everyone assumes are safe.
"I won't lock the door coz attackers might be able to bring battering ram"
1 reply →
> They tried to explain that they were running a special test and needed to shut things down before leaving, but the security guys weren't having it and made them leave right away. The test ended up running overnight, and the managers came in the next morning to over $100,000 worth of damage to the refrigeration equipment.
Dear god I hope the moron manager responsible for that got their bonus revoked
Of course not. The security team was likely blamed, or the techs doing the test. All of the grunt labor should be ready for stupid management decisions on a moments notice and plan accordingly.
> These systems are internet connected and remotely controllable because [reasons]
I understand why it's beneficial to have these systems connected to a network. You have not explained why that network has to be the Internet, and TFA is one reason why it shouldn't be.
Well, many of them were still on dial-up modems in 2008.
[dead]
> These systems are internet connected and remotely controllable because the managers need to be able to see shifts in usage over time and take action to save electricity and provide better food storage.
That doesn't really make a ton of sense. Stores have personnel in them pretty much all the time, who can take action when necessary. Data collection for monitoring, forecasting, etc. is a reasonable thing to do obviously, and centralized aggregation clearly makes sense. But remote control smells more like a fanciful shiny tech bauble than an actually useful, let alone necessary, feature. It's easy to imagine how that feature got sold internally within an engineering organization, but it's very hard to imagine customers clamoring for it.
Also unless managers in grocery store chains means something else than in any other corporation, I somehow doubt it's managers who monitor or "take action" on remotely-controlled fridges.