Comment by ngneer
1 month ago
All of what you wrote is mostly true, except that "not 100% perfect in a number of fields and usecases" is quite an understatement. You mention the cybersecurity vertical. As a datapoint, I have put the simplest code security analysis question to ChatGPT (4o mini, for those who might say wait until the next one comes out). I made a novel vulnerable function, so that it would have never been seen before. I chose a very simple and easy vulnerability. Scores of security researchers in my vicinity spotted the vulnerability trivially and instantly. ChatGPT was more than useless, failing miserably to perform any meaningful analysis. The above is anecdotal data. Could be that a different tool would perform better. However, even if such models were harnessed by a startup to solve a specific problem, there is absolutely no way for present capabilities to yield a 30-50% HC reduction in this subdomain.
I agree. Foundational models suck at the high value security work that is needed.
That said, the easiest proof-of-value for foundation models in security today is automating the SOC function by auto-generating playbooks, stitching context from various signal sources, and being able to auto-summary an attack.
This reduces the need for hiring a junior SOC Analyst, and is a workflow that has already been adopted (or is in the process of being adopted) by plenty of F500s.
At the end of the day, foundational models cannot reason yet, and that kind of capability is still far away.
4o mini is a weak, old model. I’m pretty sure even regular o1 (non-pro) would be able to crack a simple vuln
I knew this comment would come.