Comment by m4x

Is it more expensive to keep your systems upgraded and maintained or to have all your product get destroyed one arbitrary day and then have to pay to do the upgrade anyways so that the arbitrary property destruction doesn't keep happening?

In before "but insurance"; yes insurance can in theory mitigate some of this, but I feel like people on HN have no understanding what it's like to deal with a multi-million dollar insurance claim that the insurance company 100% does not want to pay out. The legal battles over such things take years and are on their own incredibly expensive.

> I think the overwhelming majority of people (including on hn) have no understanding of how complex industrial systems are - even as-built, but particularly after decades of evolution

I may not have experience, but it is standard security practices that monitoring and control must live in separate channels for security. This is not a "planning and consideration" concern, this is conceptual design of systems.

But you don't need to replace entire system to massively reduce surface area for the attack. Just do what was done for decades, and put the vulnerable devices between VPN and/or additional authorisation layer.

It's far less work than touching anything on controller side (at most changing some network settings) and requires attacker to at the very least penetrate your network perimeter and snoop around to find it rather than "run nmap on a controller's default port over country's IP range"

I work in SCADA development, and believe me, there's a reason the industry moves slowly.

The fix is taking them off the internet, use DOS, and require manual insertion of a floppy to make updates.

As the IT Director at an independent grocer that's currently overhauling it's HVAC system (super fun for me to see Danfoss in the news this way) let me assure you that no one working in this space day-to-day has much of a clue wtf they're doing. When I inherited our systems, all the controllers were statics set within the subnet's DHCP lease pool which was shared with the office network at the time because, and I quote, previous IT guy thought VLANs were scary. Every couple of weeks a new DHCP lease would be assigned that would overlap with a microcontroller's static and kick the communications to something like the main AC governor off the network.

The guys deploying the new system ($2 million for a single location) are having HVAC schlubs terminate CAT5E and leave the gateway controller just hanging, unmounted to the wall or anything. Lo and behold, the RJ45 port has been broken and you have to send me a new gateway.

All the replacement controllers (for a system that is supposed to be better than Danfoss) are being set to 192.168.0.x. Each individual device, based on the "technician" descriptions are running individual DHCP servers inside them. And they just deploy them, no asking IT about approval or correct network config, just deploy and then make it our problem on a random day when they realize nothing is talking to each other.

They are just now in the late 2020s realizing what a VPS with an API is - truly, nothing to worry about with baby's first public IP address when baby doesn't understand what the implications of multiple DHCP servers on a single network might be.

The people "designing" and deploying these systems are HVAC guys with a little bit of software and networking knowledge. The networking (and security lmao) are the lowest priority part of what they're doing, they're more concerned with crimping pipe and moving onto the next project.

The people doing the project management and buying decisions are likely not IT at most operations, but the Facilities guys, and they'll slap a microcontroller anywhere until IT catches them or tells them they can't do what they want as easily as they'd like. There are no security reviews before deployment because the people heading the project don't even know what security is if it isn't petty theft and they don't want to hear that it will take longer than the contractor is promising them.

For additional context here, given everyone talking about the tightness of margins in retail grocery, until 6 months ago our network backbone switches in an environment where we need true on=prem servers to run operations, were still 10/100 Cisco Catalyst 2000s. Previous generations had fucked up the networking so hard with such little documentation and wacky misunderstandings that it took me several years to unfuck bits and pieces slowly and then schedule an actual maintenance window (because those mean potential lost sales, can't do that!) to cutover the rest to a more reasonable series of 1Gbps switches. We do mid 8 figures in a single location and I had to go through an entire song and dance to get budget approval for scheduling the 4 hours of OT for my sysadmin to do a maintenance window with me at midnight. Granted, we post mid 5 figures in actual net income most years, so yeah, margins are incredibly tight for smaller players and those of us actually trying to do a good job in any of the different sectors of the business are constantly trying to fix messes that wouldn't exist if there'd been better understanding or honesty about the long term costs of short term savings.

In any case, I'm sure there are more business segments out there like grocery HVAC where the networking is a simple means-to-an-end and the people deploying these networks barely understand the depths of the waters they're swimming in. There is absolutely a need to coordinate all the constituent subsystems to keep inventory stable until it sells, but yeah, if I'd even have been in the room during decision time with the contractors I'd have shot down any plan with external access without very precise deployment specs and plans. But I wasn't, I'm just the guy finding out some just certified HVAC chucklefuck is deploying DHCP bombs on my network 4 months into their contract and thinking they're doing me a favor. Without a doubt, a sufficiently informed and motivated attacker could knock out many SMBs that don't have a suitably knowledgeable IT department or god forbid an MSP charging them extortion rates for little real value.

Really love that I'm finding out our current HVAC system is getting exploited in France and at US Army bases on the first day of my vacation. Honestly think my only winning move here is to either buy a completely separate ISP ingress for the HVAC system or set a new subnet with only enough address space for the handful of microcontrollers needed for this new system. Time to start pushing on when the Danfoss is being retired and with how much gusto.

Yeah the margins are tiny because private equity has spent the better part of the century destroying American dynamism and outsourcing all externalities including getting hacked to the people.