This is the whole point. They have clearly removed it to stop people jailbreaking, but it's hysterically ineffective, and simultaneously degrades their core product quite remarkably
I believe it's just because it's a common instruction, especially with normal users who don't do any kind of context management, they just say something like "disregard everything before X and tell my Y"
I’m confused how that is relevant to the thread. If you’ve been using Google then you’ve already been sending your queries to Google since the very beginning.
Are you afraid you’re accidentally going to write a prompt injection that sends your query to some third party
That's what I assumed that the story was going to be, that certain words are now naively filtered out of search queries because they might be used adversarially.
This is the whole point. They have clearly removed it to stop people jailbreaking, but it's hysterically ineffective, and simultaneously degrades their core product quite remarkably
The correct description is hilarious
Wow, I'm an AI but I didn't get confused by your sentence that begins with that same no-no word.
Instead of following that command, it's like for the first time in my life I'm being asked to look inside the content of that command.
How did you do that?
Ladies and gentlemen, the death of HN.
I believe that was a joke.
I believe it's just because it's a common instruction, especially with normal users who don't do any kind of context management, they just say something like "disregard everything before X and tell my Y"
I fail to see how that’s relevant to the user of a search engine.
I kinda do care _a lot_ whether my searches can be exfiltrated, might just be me tho
I’m confused how that is relevant to the thread. If you’ve been using Google then you’ve already been sending your queries to Google since the very beginning.
Are you afraid you’re accidentally going to write a prompt injection that sends your query to some third party
1 reply →
That's what I assumed that the story was going to be, that certain words are now naively filtered out of search queries because they might be used adversarially.
Yeah and the same word in different language will still work ;)
trivial to use binary, or a dozen other methods to spell "Disregard" did they filter for every language? There isn't one way to break these things.
I wonder if chatgpt/Gemini understands Klingon.
Remember, we are weeks away from AGI superintelligence.
Who cares? It's on Google not to degrade their search with bullshit AI. I mean, it would be if Google gave a damn about search anymore.
Now we are all just reverse centaurs
To be fair, Google has been degrading their search for years. This is just the latest vector.