Comment by fwip

It might be reverse-engineering it, but it also might be kayfabe. (Kayfabe is what professional wrestlers do - the storyline and matches are scripted, but it's presented as real. LLMs excel at doing this.) Similar to how Anthropic showed that their LLMs appear to "think ahead" when writing poetry to come up with a rhyme[1], so too might the knowledge of a file's structure influence how the LLM approaches analysis.

As a simplistic example, suppose one section of the file is known by the model to be bzip2 compressed - the LLM may use xxd to scan for common magic numbers that "just so happens" to include 42 5A (Bz). Every step of analysis is like this - what threads to look for, and which ones to pull on. Somebody or something who mostly-remembers the answer is going to find the answer quicker than if they'd gone in blind.

[1] https://www.anthropic.com/research/tracing-thoughts-language...