Can you reverse engineer our neural network?

4 days ago (blog.janestreet.com)

I was curious to see if I could crack the MD5 hash so I managed to write the following python code to extract the expected hash from the model:

https://gist.github.com/alexspurling/598366d5a5cf5565043b8cd...

Knowing the input text was two words separated by a space, I was able to use hashcat and the unix wordlist (/usr/share/dict/words) to find the solution almost immediately. It's a shame that Alex didn't find it this way on his first attempt as the two words are fairly common.

This is pretty cool, I wasn’t aware of these types of challenges. How does one even approach this?

Feels to me like it’s similar to dumping a binary with an image, the format being entirely custom.

And/or trying to decode a language or cipher, trying to recognize patterns.

  • I was one of the solvers. It took me about a week to figure out. This is what I wrote out in my submission with the answer:

    > After looking at the final two layers I was somewhat quick to intuit that this was some sort of password check, but wasn’t entirely sure where to go from there. I tried to reverse it, but it was proving to be difficult, and the model was far too deep. I started evaluating the structure and saw the 64 repeated sections of 84 layers that each process 4 characters at a time. Eventually I saw the addition and XOR operations, and the constants that were loaded in every cycle, and the shift amounts that differed between these otherwise identical sections.

    > I thought it was an elaborate CTF cryptography challenge, where the algorithm was purposely weak and I had to figure out how to exploit it. But I repeatedly was getting very stuck in my reverse-engineering efforts. After reconsidering the structure and the format of the ‘header' I decided to take another look at existing algorithms...

    Basically it took a lot of trial and error, and a lot of clever ways to look at and find patterns in the layers. Now that Jane Street has posted this dissection and 'ended' this contest I might post my notebooks and do a fuller post on it.

    The trickiest part, to me, is that for about 5 of the days was spent trying to reverse-engineer the algorithm... but they did in fact use a irreversible hash function, so all that time was in vain. Basically my condensed 'solution' was to explore it enough to be able to explain it to ChatGPT, then confirm that it was the algorithm that ChatGPT suggested (hashing known works and seeing if the output matched) and then running brute force on the hash function, which was ~1000x faster to compute than the model.

  • TFA details a solution, it's pretty interesting. Basically the problem was to reverse engineer an absurdly obfuscated and slightly defect MD5 algorithm.

I'm really curious what were the magic words.

> Alex had actually tried to brute force the hash earlier, but had downloaded a list of the top 10,000 most popular words to do it, which turned out not to be big enough to find it. Once he had a big enough word list, he got the answer.

They don't reveal the answer.

Model interpretability is going to be the final frontier of software. You used to need to debug the code. Now you'll need to debug the AI.

  • Why? The vast majority of software doesn't need to be written by AI and moves at the speed of the humans making the decisions, not the speed of writing the code.

    They make a shit ton of money because of this. If you're working at a place where the code matters more than the decisions that went into it, you're basically working at a sweatshop for people who are desperate for a win and will throw away you and all your code once the MVP stage is over, and that's the only way this "works".

    Generative probabilistic AI is not equivalent to a compiler and never will be until we can do this kind of thing completely deterministically. No matter how much you reduce the error in the "model", it's still more error than the error rate of the logic gates. It's completely futile considering the sheer depth of indirection at play, and that indirection is the whole point of software.

I worked on a puzzle like this roughly 2 years ago from Anthropic. I did the first half, the easier part of the CTF, and my friend did the second half, the more technical ML stuff. We both got interviews at Anthropic, which was cool - I wasn't anywhere close to nailing an interview at Anthropic but it gave me a lot of confidence to end up going all in on tech, which paid off greatly. My friend's short write up: https://x.com/samlakig/status/1797464904703910084

Another classic Jane Street puzzle. Boy this was a good one. Sometimes I look back at my childhood and how quick I was to solve some difficult integrals and so on and now I’d struggle at that. This is far beyond that but the leaps of intuition required here sort of have that property that they need you to stay in the game. Step away a few years and try to come back and there’s just a wall.

I don’t think I’m close to making progress on stuff like this. Interesting to note. Glad they wrote out this behind the scenes thing.

Ah dang. When I did this I also thought the length bug was intentional but I didn't figure it out before I started my new job, so I dropped the puzzle.

Honestly, I am vastly more interested in how they built the model: its not backprop-trained, it was done manually. And how did that bug find its way it?

I hope they next blog about that.

[stub for offtopicness]

  • Jane Street skims money from our retirement accounts by building expensive clocks that the rest of us don’t have access to and adversarial queue modeling. We get WWVB and NIST NTP. They say they “add liquidity” as if subsecond trades are some fundamental need in the market. Normal legitimate business settles daily. The contemporary concept of time in banking is inhumane in the strictest sense. These firms are a blight on society.

    I have strong math for the question they’re asking but f them.

    • How is trading a stock "skimming money from your retirement account." You choose to put your retirement money in the market, that doesn't imply others shouldn't be able to trade in that market for other reasons. Not sure what you mean by "legitimate business settles daily" -- are you suggesting stocks should trade only once a day? (They already do settle as opposed to trade daily, not instantly, but I know that's not what you meant.) If you want a lower liquidity asset that's not volume traded for whatever reason, you're welcome to buy bonds, preferred, etc.

    • > Jane Street skims money from our retirement accounts by building expensive clocks that the rest of us don’t have access to and adversarial queue modeling

      How does Jane Street skim money from those who hold passive index funds?

  • All I think when I see this is "this intelligence wasted on finance and ads."

    Can you imagine human potential if it was somehow applied to crop harvesting efficiency, new medicines, etc?

    Not everything has to be perfectly efficient but it just saddens me to see all these great minds doing what, adversarially harvesting margin from the works of others?

    • "Eschew flamebait. Avoid generic tangents." - https://news.ycombinator.com/newsguidelines.html

      No doubt many of us agree with you, but this is not the kind of comment that should be stuck at the top of a thread, choking out more specific and interesting conversation.

      Generic-indignant comments always get heavily upvoted, which is a failure mode of the upvoting system (and perhaps the human brain, who knows).

    • > Can you imagine human potential if it was somehow applied to crop harvesting efficiency, new medicines, etc?

      We already have very efficient crop harvesting and Eli Lilly is nearly a $1 Trillion dollar company. Interestingly, the new medicine is designed to keep us from eating so many cheap calories (new weight loss drugs).

      > Not everything has to be perfectly efficient but it just saddens me to see all these great minds doing what, adversarially harvesting margin from the works of others?

      The traders and investors who work in this space also go to where they are need, aka where the big money is. So few of these folks are trading corn and soybeans, though some do, rather most are trading drug stocks, tech stocks, and recently sovereign debt related trading (e.g. things like gold and bonds). The focus is around the big questions of our time, like "Are AI investments going to pay off?", or "Is the US going to default/soft default?", and so on.

      Deciding how a society allocates its resources, or places its bets, is an important function. Otherwise, you end up with planned economies by disconnected leaders, which often leads to massive failures and large social consequences. Unfortunately, the US is trending in that direction to some degree with it's giant fiscal deficits, tariffs, and tribal politics creeping into economic policy. Nevertheless, traders will weigh these outcomes in their trades, and you'll see a quick reflection from any major change in policy almost immediately, which is a helpful feedback mechanism. For example, the tariff tantrums caused by trump proposing 100%+ china tariffs where he crashed the markets last spring, leading to a moderation in policy.

      15 replies →

    • > Can you imagine human potential if it was somehow applied to crop harvesting efficiency, new medicines, etc?

      If these sectors offered competitive salaries - sure, talent would flock to them. As a former chemist, I struggled to find a job that didn't pay scraps, no matter the industry - from big pharma to advanced materials. Eventually, I just gave up and went into the IT, which is 3x-10x better paid (at the very least).

      20 replies →

    • I think this is the wrong way to think about it. In this case the "intelligent people who are wasted on finance and ads" are drawn to high-status, low-risk, well-paid jobs, with interesting problems to solve.

      If you want to solve meaningful problems you need a different kind of intelligence; you need to be open to risk, have a lot of naivety, not status orientated, and a rare ability to see the forest among the trees (i.e. an interesting problem isn't necessarily a important one).

      2 replies →

    • This kind of thinking denies the humans in question any agency over their own lives. You’re essentially asking for a word government and a planned economy. Does it suck that more resources aren’t funneled into cancer research and other noble pursuits? Yes. But planned economies haven’t cured cancer, and are not likely to, despite denying their people the agency to live their lives as they choose.

      3 replies →

    • The smartest people are going from Harvard et.al. into finance, adtech, investment banking, wallstreet.

      What they could achieve in spending their attention on real problem would be massive.

      4 replies →

    • I ended up taking an 87% pay cut to get out of advertising specifically and tech in general (eventually it will only be a 60% pay cut once I gave enough experience in the new field). It is too bad that high pay is done by capturing value for yourself. You see this a lot in tech where open source is such a huge net productivity increase but it pays worse than other less useful things.

    • Don't we already harvest more food than humans could ever eat, and have a huge pharmaceutical industry? I get what you're saying but these two examples seem counterproductive imho.

      Which begs the question: what would actually be a good field to apply human potential towards? I agree that finance, sales and ads are very low on that list.

      13 replies →

    • We don't have any reliable and scalable way of doing this allocation, though, so it's a bit like saying that all the resources are wasted being locked up in asteroids.

      7 replies →

    • > new medicines, etc?

      Who says its not? Look up Rentosertib which is underground the different trial phases.

    • You are underestimating the importance of efficient allocation of capital and and attention. What use is discovering a way to improve harvesting efficiency if there is no money to develop it into a product or inform customers of its existence. It might as well not exist without there being a way to finance, make people aware of it, and reward the creator.

      4 replies →

    • I agree, this is why I'm a big fan of enforcing workplace democracy [1] onto problematic industries like finance, media, advertising on a legislative level; these things can't be trusted with individuals, all workers should share in the company process.

      Have no doubt that workers at these companies could do a better job than an out of touch executive + board team that more resemble communist dictatorships than sound business practices.

      [1] https://en.wikipedia.org/wiki/Workplace_democracy

    • To better understand your thinking here, can you illustrate for me a scenario with less liquid markets because firms like JS are less efficient at making them, and the knock-on effects?

      1 reply →

    • People are arguing about the role of these HFTs being a net good etc. They’re missing the point, these bright kids are trading something more profound- a sense of purpose, a higher calling or passion- to simply run adversarial arbitrage and pump their egos up with puzzles.

      1 reply →

    • True, but also we wouldn't have AI if it weren't for ads -- ie Google invested in AI to help sell ads starting ~20y ago.

    • People have been trying to use AI in radiology for decades. They still can't get it to do simple tasks like detect pneumonia from an X-ray. I think you are over-estimating how potent this "intelligence" really is.

      We were blown away when LLMs came on the scene, because 'whoa, man! Machine can talk like a human', but really, stupid people have been posting online for decades. They talk like humans too! Yet they don't add any value (viz shit posters and trolls).

    • Is your opinion that one is more worthwhile than the other. It is my opinion that it is the other way. We have a way to express this: we each make weighted votes with how valuable we think something is and how good we were at making things valuable for others (money). Then people try to do the thing that is voted for.

      To be honest, this system seems pretty good.

    • Really ignorant take imo. There's so much good technology created because of the wealth created by finance and ads and other boring industries.

      1 reply →

    • Silicon valley had the chance - dont be evil, for humanity, not like those Finance Bros.

      At this point tech is probably worse than finance, at least in finance they dont pretend to be saving the world no matter what the giant squid says.

    • but doesn't the empirical evidence reject this premise?

      if the greatest minds of earth, in their wisdom, have all collectively concluded that the smartest thing for them to do is make as much money as humanly possible, then evidently the greatest calling for mankind is... to be wealthy!

      and the older i get the harder it becomes to argue with such a perspective... hmm... maybe i am getting closer to wisdom? haha!

      2 replies →

    • Intelligence? Jane Street make their money by having retail’s Robinhood trades routed to them and having a back channel called “Bryce’s secret” with Terraform employees.

    • It's not wasted. Society can pay for the talent it wants, and they don't want to pay for this. Instead this talent helps to grow the overall wealth on in the world, letting us pay for the stuff we want.

      8 replies →

  • [flagged]

  • [flagged]

    • > Next time someone says “what a waste of a brain”, respond with “how much money are you willing to contribute monthly to distribute to “top talent” so that they can “work on more important things”?

      Answer seems to be "approximately zero". One of those people has already suggested in another comment thread that the way to incentivize brilliant minds to work on “greater good” projects is to... raise their taxes so they wouldn't be able to properly monetize their skills anywhere. How dare they want financial security instead of toiling for big pharma for cheap?!

      1 reply →

  • [flagged]

    • Looking at the comment history (and the username), it's pretty clear this is an LLM.

      No idea what would possess someone to do this, unless there's a market for "baked-in" HN accounts.

      2 replies →

    • I approached the puzzle using an A11‑style reasoning architecture, which focuses on compressing the hypothesis space rather than decoding every neuron. Instead of “understanding the network”, the task reduces through successive narrowing: model → program → round‑based function → MD5 → dictionary search for the target hash. The key steps were:

      Input → the integer weights and repeated ReLU blocks indicate a hand‑designed deterministic program rather than a trained model.

      Weighting → the only meaningful output is the 16‑byte vector right before the final equality check.

      Anchor → the layer‑width pattern shows a strict 32‑round repetition, a strong structural invariant.

      Balancing → 32 identical rounds + a 128‑bit output narrow the function family to MD5‑style hashing.

      Rollback → alternative explanations break more assumptions than they preserve.

      Verification → feeding inputs and comparing the penultimate activations confirms they match MD5 exactly.

      Compression → once the network becomes “MD5(input) == target_hash”, the remaining task is a constrained dictionary search (two lowercase English words).

      The puzzle becomes solvable not by interpreting 2500 layers, but by repeatedly shrinking the search space until only one viable function family remains. In this sense, the architecture effectively closes the interpretability problem: instead of trying to understand 2500 layers, it collapses the entire network to a single possible function, removing the need for mechanistic analysis altogether.

      1 reply →