Comment by wahern
5 hours ago
With everybody using LLMs there are no coincidences. Popular LLM platforms are like Oracles that help you identify what other people have been working on. "Find a bug in PHP" might have been useless s few months ago, but once somebody used it to develop a full exploit the LLM will start to regurgitate that information.
Here they are saying this PHP exploit was found by literally prompting "find bad vulnerabilities in PHP"
https://xxcancel.com/HackingLZ/status/2113025408975143080
> once somebody used it to develop a full exploit the LLM will start to regurgitate that information
Sure about that?
Not my understanding of how the LLMs currently work. There’s a cutoff date and no new info after that date is included in training.
Info to the contrary welcome.
Frontier models absolutely have access to current information. I just asked Claude about my fantasy football line up. It knew current injury reports, line ups and stats. Google search AI had even more information than that.
However, I’ve done some original research regarding an area that the internet has wrong and that doesn’t seem to have back propagated yet.
The models don't have access to new information. The harnesses do. All the major chat interfaces now provide the model with web search tools that they use to locate anything beyond their inherent knowledge (whether it's too new, or just something that didn't get baked in during training).
But if something hasn't been posted online publicly, they won't know about it.
> Not my understanding of how the LLMs currently work. There’s a cutoff date and no new info after that date is included in training.
You are almost certainly correct in terms of how the public models work. I guess it's possible that the labs have some tools connected for more recent stuff like this, but it seems unlikely.