Comment by nickpsecurity
16 hours ago
The solution to this in the Orange Book (TCSEC) days in the 1980's-1990's was a system fully traceable from requirements to code, proven to embed a security policy, and analyzable and buildable from source locally by the customer using existing, trusted tools. Eventually, people added hashes for the code and data.
So, your program that combines source files or checks dependencies would be fully specified in its success and failure states. Only combinations of functions leading to a provably-secure state are even allowed. If you can't do that, the feature is too complex to allow. Human pentesters review it from design to algorithms to building it to spot ways attacks might happen.
That's what it takes to build software that usually resists subversion. Most software isn't built that way. It can't be because the priorities of developers and customers work against it. So, we'll continue to see clever attacks that exploit systems not designed to high security standards.
For this topic, I recommend David A. Wheeler's page on Software, Configuration Management Security because it covers many issues with it in mostly-centralized systems.
What happens when "security" includes a time component? A shocking number of modern systems depend on time, either in enough time having elapsed to prove something about the attacker, or in little enough time elapsing as a critical component of the system in question. That feels like it escapes the bounds of your definitions and is also somewhat unavoidable. Is that rectifiable somewhere?
> Most software isn't built that way. It can't be because the priorities of developers and customers work against it.
The most significant such priority may be the costs of paying developers and of time to delivery.
The dramatic reductions in those costs due to LLMs enable us to produce much more quantity and/or quality. Many complain now about quantity, so perhaps we are finally at a stage where we don't need much more software, and can focus on quality. Also, LLM attackers create demand for higher quality.
In other words, LLMs might enable us to some of these things that were impossible before.
It's very exciting to see the work on combining AI models with static analysis, test generation, formal proof, and refactoring. All of these suggests we might see high assurance (EAL6+) developed rapidly in the future. At least for combinations of well-understood concepts.
You don't mention generating code from scratch; is that on purpose? Perhaps a developer could add many constraints that would have been too expensive previously - e.g., code in Spark, use only these ___ system calls, never use this riskier capability, use only these ___ libraries, etc.
The LLM might take longer than otherwise, but still be fast enough. The new technology, like every useful one, changes the tradeoff equation.