Comment by red-iron-pine

same experiences. lots of talk but most AI was locked down -- for good reasons, probably -- and we have explicitly allowed platforms.

in practice copilot is only useful for advanced search and it is really only useful for surface answers. Was a time saver for researching common tech I'm not strong in, like if certain Cisco platforms support X, or if there are simple ways to do Y in MongoDB, etc.

for example we had it search for 9-10 CVEs to see if any were exploited in the wild. Copilot got most of them wrong. ChatGPT got most of them right... except for one or two. But now I'm not sure I can trust anything and I'm checking them all from scratch anyway.

Both were confident in ways my security intern was not, and my intern was able to tell me "I have no idea about [CVE]".