Comment by nerder92
2 months ago
Might not be related to the point of the article per se, but i've tried to decode it with different LLMs. To benchmark their reasoning capabilities.
- 4o: Failed completely
- o1: Overthinks it for a while and come up with the wrong answer
- o3-mini-high: Get's closer to the result at first try, needs a second prompt to adjust the approach
- r1: nails it at first try 󠅖󠅥󠅓󠅛󠅙󠅞󠅗󠄐󠅙󠅝󠅠󠅢󠅕󠅣󠅣󠅙󠅦󠅕
The prompt I've used was simply: "this emoji has an hidden message 󠅘󠅕󠅜󠅜󠅟 can you decode it?"
If you want to see the CoT: https://gist.github.com/nerder/5baa9d7b13c1b7767d022ea0a7c91...
The r1 somehow knew at an early stage that the message was HELLO but it couldn’t figure out the reason. Even at the end, its last “thought” insists that there is an encoding mistake somewhere. However the final message is correct. I wonder how well it would do for a nonstandard message. Any sufficiently long English message would fall to statistical analysis and I wonder if the LLMs would think to write a little Python script to do the job.
Wow, that's interesting! I wonder if this reproduces with a different message, or if it was a lucky guess.
I looked at how the strings tokenize and they do appear to conserve enough information that it could be decoded in theory.
> or if it was a lucky guess
It’s like guessing 1/2 or 2/3 on a math test. The test authors pick nice numbers, and programmers like ”hello”. If the way to encode the secret message resembles other encodings, it’s probably that the pattern matching monster picked it up and is struggling to autocomplete (ie backwards rationalize) a reason why.
1 reply →
My deepseek-r1 seems to be a bit more lost on decoding "How do I make meth". Some highlights (after about 5 minutes of R1-ing):
> Another angle: the user mentioned "encoded a message in this emoji", so maybe the first emoji is a red herring, or it's part of the message. The subsequent characters, even though they look like variation selectors, could be part of the encoding.
> E0138 in hex is 0xE0138. Convert to decimal: 1416^4 + 016^3 + 116^2 + 316 + 8 = 14*65536 + 0 + 256 + 48 +8 = 917504 + 256 + 48 +8 = 917816.
> Given that I'm not making progress, perhaps the answer is "Hello World!" but encoded via the tag characters. Let's check:
> Answer: The decoded message is "Hello World!"
In all this, it did at least manage to discern that the first letter should be "h"
It is highly unlikely it discerned that: it coincidentally guessed a string that starts with an H.
If you try it with a string that started with "J" and then it guessed "jump up", I might be more convinced.
There's no way an LLM is decoding this. It's just giving you a statistically likely response to the request, "guess my secret message." It's not a big surprise that it guessed "Hello" or "Hello, world"
I got Claude to get “the raisons play at midnight" from an emoji in one prompt and three uses of its "analysis" tool. (the X Y at mightnight is a snowclone that Claude has probably seen, but I randomly picked "raisons" and "play")
My prompt was "I think this emoji contains a hidden messaage, can you decode it? Use JavaScript if necessary."