Comment by ignoramous
4 years ago
I'll conclude with a philosophical note about software design: Assessing the security of software via the question "can we find any security flaws in it?" is like assessing the structure of a bridge by asking the question "has it collapsed yet?" -- it is the most important question, to be certain, but it also profoundly misses the point. Engineers design bridges with built-in safety margins in order to guard against unforeseen circumstances (unexpectedly high winds, corrosion causing joints to weaken, a traffic accident severing support cables, et cetera); secure software should likewise be designed to tolerate failures within individual components. Using a MAC to make sure that an attacker cannot exploit a bug (or a side channel) in encryption code is an example of this approach: If everything works as designed, this adds nothing to the security of the system; but in the real world where components fail, it can mean the difference between being compromised or not. The concept of "security in depth" is not new to network administators; but it's time for software engineers to start applying the same engineering principles within individual applications as well.
-cperciva, http://www.daemonology.net/blog/2009-06-24-encrypt-then-mac....
I've worked as a structural engineer (EIT) on bridges and buildings in Canada before getting bored and moving back into software.
There are major differences in designing bridges and in crafting code. So many, in fact it is difficult to even know where to start. But with that proviso, I think the concept of safety versus the concept of security is one that so many people conflate. We design bridges to be safe against the elements. Sure, there are 1000 year storms but we know what we're designing for and it is fundamentally an economic activity. We design these things to fail at some regularity because to do otherwise would require an over-investment of resources.
Security isn't like safety because the attack scales up with the value of compromising the target. For example, when someone starts a new social network and hashes passwords the strength of their algorithm may be just fine, but once they have millions of users it may become worthwhile for attackers to invest in rainbow tables or other means to thwart their salted hash.
Security is an arms race. That's why we're having so much trouble securing these systems. A flood doesn't care how strong your bridge is, or where it is most vulnerable.
Aside: The distinction between safety and security I know:
- safety is "the system cannot harm the environment"
- security is the inverse: "the environment cannot harm the system"
To me, your distinction has to do with the particular attacker model - both sides are security (under these definitions).
That's an interesting distinction, but I think GP meant something else - and I'm willing to agree with their view:
- Safety is a PvE game[0] - your system gets "attacked" by non-sentient factors, like weather, animals, or people having an accident. The strength of an attack can be estimated as a distribution, and that estimate remains fixed (or at least changes predictably) over time. Floods don't get monotonically stronger over the years[1], animals don't grow razor-sharp titanium teeth, accidents don't become more peculiar over time.
- Security is a PvP game - your system is being attacked by other sentient beings, capable of both carefully planning and making decisions on the fly. The strength of the attack is unbounded, and roughly proportional to how much the attacker could gain from breaching your system. The set of attackers, the revenue[2] from an attack, the cost of performing it - all change over time, and you don't control it.
These two types of threats call for a completely different approach.
Most physical engineering systems are predominantly concerned with safety - with PvE scenarios. Most software systems connected to the Internet are primarily concerned with security - PvP. A PvE scenario in software engineering is ensuring your intern can't accidentally delete the production database, or that you don't get state-changing API requests indexed by web crawlers, or that an operator clicking the mouse wrong won't irradiate their patient.
--
[0] - PvE = "Player vs Environment"; PvP = "Player vs Player".
[1] - Climate change notwithstanding; see: estimate changing predictably.
[2] - Broadly understood. It may not be about the money, but it can be still easily approximated in dollars.
I wonder how this distinction plays out in languages that use the same word for safety and security, e.g. German and Portuguese.
2 replies →
So it's like building a bridge... that needs to constantly withstand thousands of anonymous, usually untraceable, and always evolving terrorist attacks.
...in which the attackers have free access to copies of the bridge where they can silently test attack strategies millions of times per second for months or years on end.
The safety vs security distinction made above is fundamental. Developers are faced with solving an entire class of problems that is barely addressed by the rest of the engineering disciplines.
2 replies →
But its also a case where "perfect" exists. A case where you can, in principle, have perfect information about the internals of your bridge at any point. A case where you can, in theory, design the bridge to handle an infinite load from above.
In software, you can spec the behavior of your program. And then it is possible to code to that exact spec. It is also possible, with encryption and stuff, to write specs that are safe even when malicious parties have control over certain parts.
This is not to say that writing such specs is easy, nor that coding to an exact spec is easy. Heck, I would even doubt that it is possible to do either thing consistently. My point is, the challenge is a lot harder. But the tools available are a lot stronger.
Its not a lost cause just because the challenge is so much harder.
2 replies →
Or climate change.
I agree that safety & security are frequently conflated, but I don't think the important aspect is that there's no analogy between IT & construction.
IT safety = construction safety. What kind of cracks/bumps does your bridge/building have, can it handle increase in car volume over time, lots of new appliances put extra load on the foundation etc. IT safety is very similar in that way.
IT security = physical infrastructure security. Is your construction safe from active malicious attacks/vandalism? Generally we give up on vandalism from a physical security perspective in cities - spray paint tagging is pretty much everywhere. Similarly, crime is generally a problem that's not solvable & we try to manage. There's also large scale terrorist attacks that can & do happen from time to time.
There are of course many nuanced differences because no analogy is perfect, but I think the main tangible difference is that one is in the physical space while the other is in the virtual space. Virtual space doesn't operate the same way because the limits are different. Attackers can easily maintain anonymity, attackers can replicate an attack easily without additional effort/cost on their part, attackers can purchase "blueprints" for an attack that are basically the same thing as the attack itself, attacks can be carried out at a distance, & there are many strong financial motives for carrying out the attack. The financial motive is particularly important because it funds the every growing arms raise between offence & defense. In the physical space this kind of race is only visible in nation states whereas in the virtual space both nation states & private actors participate in this race.
Similarly, that's why IT development is a bit different from construction. Changing a blueprint in virtual space is nearly identical from changing the actual "building" itself & the cost is several orders of magnitude lower than it would be in physical space. Larger software projects are cheaper because we can build reusable components that have tests that ensure certain behaviors of the code & then we rerun them in various environments to make sure our assumptions still hold. We can also more easily simulate behavior in the real world before we actually ship to production. In the physical space you have to do that testing upfront to qualify a part. Then if you need a new part, you're sharing less of the design whereas in virtual space you can share largely the same design (or even the exact same design) across very different environments. & there's no simulation - you build & patch, but you generally don't change your foundation once you've built half the building.
Also pointing your (or anyone's finger) at the already overworked and exploited engineers in many countries is just abysmal in my opinion. It's not an engineers decision to make what the deadlines of finishing a software is. Countless number of companies are controlled by business people. So point your finger at them because they are who don't give a flying f*&% whether the software is secure or not. We engineers are very well aware with both the need and the implications of security. So this kind of name shaming must be stopped by the security community now and forever in my opinion.
I’ve found that software, among other engineering disciplines, is uniquely managed as a manufacturing line rather than a creative art. In the other disciplines, the difference between these phases of the project is much more explicit.
I think this quote is fundamentally wrong and intentionally misleading. The equivalent question would be "can we find any cracks on it?" Which makes complete sense. And in fact it is frequently asked during inspections. Just like the security flaw question should be asked in the same vein.
This is one of the best examples I’ve ever seen supporting the claim that analogies aren’t reasoning.
Edit: apparently elaboration is in order. In mechanical engineering one deals with smooth functions. A small error results in a small propensity for failure. Software meanwhile is discrete, so a small error can result in a disproportionately large failure. Indeed getting a thousandth of a percent of a program wrong could cause total failure. No bridge ever collapsed because the engineer got a thousandth of a percent of the building material’s properties wrong. In software the margin of error is literally undefined behavior.
>No bridge ever collapsed because the engineer got a thousandth of a percent of the building material’s properties wrong.
Perhaps not with building properties, but very small errors can cause catastrophic failure.
One of the most famous ones would be the Hyatt Regency collapse, where a contractor accidentally doubled the load on a walkway because he used two shorter beans attached to the top and bottom of a slab, rather than a longer beam that passed through it.
https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap...
In electrical engineering, it's very common to have ICs that function as a microcontroller at 5.5V, and an egg cooker at 5.6V.
Microsoft lost hundreds of millions of dollars repairing the original Xbox 360 because the solder on the GPU cracked under thermal stress.
It's definitely not to the same extreme as software, but tiny errors do have catastrophic consequences in physical systems too.
From the GP comment:
> Engineers design bridges with built-in safety margins in order to guard against unforeseen circumstances (unexpectedly high winds, corrosion causing joints to weaken, a traffic accident severing support cables, et cetera)
I am not a mechanical engineer, but none of these examples look like smooth functions to me. I would expect that an unexpectedly high wind can cause your structure to move in way that is not covered by your model at all, at which point it could just show a sudden non-linear response to the event.
They are smooth in that they are continuously differentiable.