Because the article shows it isn't Gemini that is the issue, it is the tool calling. When Gemini can't get to a file (because it is blocked by .gitignore), it then uses cat to read the contents.
I've watched this with GPT-OSS as well. If the tool blocks something, it will try other ways until it gets it.
How can an LLM be at fault for something? It is a text prediction engine. WE are giving them access to tools.
Do we blame the saw for cutting off our finger?
Do we blame the gun for shooting ourselves in the foot?
Do we blame the tiger for attacking the magician?
The answer to all of those things is: no. We don't blame the thing doing what it is meant to be doing no matter what we put in front of it.
Because it misses the point. The problem is not the model being in a cloud. The problem is that as soon as "untrusted inputs" (i.e. web content) touch your LLM context, you are vulnerable to data exfil. Running the model locally has nothing to do with avoiding this. Nor does "running code in a sandbox", as long as that sandbox can hit http / dns / whatever.
The main problem is that LLMs share both "control" and "data" channels, and you can't (so far) disambiguate between the two. There are mitigations, but nothing is 100% safe.
Sorry, I didn't elaborate. But "completely local" meant not doing any network calls unless specifically approved. When llm calls are completely local you just need to monitor a few explicit network calls to be sure.
Because the article shows it isn't Gemini that is the issue, it is the tool calling. When Gemini can't get to a file (because it is blocked by .gitignore), it then uses cat to read the contents.
I've watched this with GPT-OSS as well. If the tool blocks something, it will try other ways until it gets it.
The LLM "hacks" you.
And… that isn’t the LLM’s fault/responsibility?
As the apocryphal IBM quote goes:
"A computer can never be held accountable; therefore, a computer must never make a management decision."
How can an LLM be at fault for something? It is a text prediction engine. WE are giving them access to tools.
Do we blame the saw for cutting off our finger? Do we blame the gun for shooting ourselves in the foot? Do we blame the tiger for attacking the magician?
The answer to all of those things is: no. We don't blame the thing doing what it is meant to be doing no matter what we put in front of it.
3 replies →
Because it misses the point. The problem is not the model being in a cloud. The problem is that as soon as "untrusted inputs" (i.e. web content) touch your LLM context, you are vulnerable to data exfil. Running the model locally has nothing to do with avoiding this. Nor does "running code in a sandbox", as long as that sandbox can hit http / dns / whatever.
The main problem is that LLMs share both "control" and "data" channels, and you can't (so far) disambiguate between the two. There are mitigations, but nothing is 100% safe.
Sorry, I didn't elaborate. But "completely local" meant not doing any network calls unless specifically approved. When llm calls are completely local you just need to monitor a few explicit network calls to be sure.
In a realistic and useful scenario, how would you approve or deny network calls made by a LLM?
2 replies →