Comment by tyleo
1 day ago
That doesn't sound right. For example, there's plenty of software with the correct observable behavior which leaks credentials. So what needs to be captured goes beyond observable behavior.
1 day ago
That doesn't sound right. For example, there's plenty of software with the correct observable behavior which leaks credentials. So what needs to be captured goes beyond observable behavior.
Leaking credentials is observable behavior.
Certainly you could write specification for a piece of software, and the software could meet the specification while also leaking credentials. Obviously, that would be a problem. But at some point, this starts to feel artificial and silly. The same software could reformat your hard disk, right?
At some point, we aren’t discussing whether or not AI is doing a bad job writing software. We’re discussing whether or not it’s actively malicious.
If leaking credentials is observable behavior. I don't get your point.
Memory leaks, deleting the hard drive, spending money would all be observable behavior.
By your reasoning that the "observable behavior needs to be specified rigorously" it seems like you'd have to list these all out. We do, after all, already have cases of AI deleting data.
That sounds harder and more error prone than what we're doing now by rigorously defining these defects out of existence in code.
My ultimate point is that source code specifies everything the software can or will do, but most of that is not part of the required specification. Most of it is implementation details you don’t care about.
The entire reason we have functions and components and modules etc is to isolate engineers from the things we do not need to care about. I should not need to care about the implementation details of most software, only if it meets my retirements.
The move to AI first software development will not happen because we find a way to specify as much in English as we previously would have specified in a programming language. The move will happen when and as we figure out how to specify the things that matter. We don’t need the same rigor. We need the correct rigor.
7 replies →