Comment by moron4hire

I think the point of whether we consider user input to be instructions or data is important and I think it should be front of mind for everyone.

But I don't agree prompt injection vs SQL injection is an example of this kind of failure, at least not in this case where it's giving unauthorized access to data. And I don't think the fix really needs to go as far as creating wholly new training methods.

That's because the LLM doesn't have access to the repositories on its own. It has to be given that access through deterministic tools programmed in traditional programming languages. Even the ability to RAG search needs a part A to perform a vector nearest neighbor clustering and part B to retrieve the data found via the embedding index, both of which the LLM can't do on its own.

Prompt injection providing access to unauthorized data is 100% lazy tool development where those tools do not operate through any form of access control. You'd have the same unauthorized access with properly parametrized SQL if none of the search inputs were the user credentials.

This is one of the major dangers of "LLMs are going to democratize coding." Software development isn't a safe field of play. Not only are there a lot of dangers, many of them are subtle, unintuitive, and quite easy to stumble upon. That's why we idealized a mentorship model for junior developers, to try to limit the blast radius of mistakes in a safe, pro-learning environment. But the ever hard driving quest to eliminate software engineers as a species is pushing people into ludicrously stupid actions like giving LLMs full access to write SQL queries and full access to operate the CLI. The problem is not that we are treating the user's input as unfiltered instructions, it's that we're forgetting that the LLM is another agent in the system and treating the LLM's input as unfiltered instructions.