Jane Street, one of the world’s most selective quantitative trading firms, published a mechanistic interpretability puzzle a few weeks ago. They hand-crafted a neural network with approximately 2,500 linear layers, integer weights, and released it to the public with a question: What function does this network compute?
The answer: MD5. A cryptographic hash algorithm from 1992, implemented entirely as matrix multiplications and ReLU functions.
What matters isn’t the answer. It’s the path the winner took to reach it. Because that path is, without exaggeration, a manual for debugging opaque systems that applies far beyond machine learning.
The experiment
This wasn’t a typical black box puzzle. Participants received the complete model specification: all weight matrices, all biases, the entire architecture. No need to guess the structure. The challenge was understanding what it did.
The network accepted a text string as input and returned 0 or 1. Their example: “vegetable dog” produces 0.
With ~2,500 linear layers, about 2 million nodes, and no documentation, the question was: what’s the relationship between input and output?
How it was solved: a debugging case study
The winner, Alex, followed a process any senior engineer would recognize. Not because of the specific tools, but because of the reasoning structure.
Phase 1: Observe before touching
Alex’s first move was to visualize the weight matrices. He didn’t run the model. Didn’t try to train it. He looked at the data.
What he saw: all weights were integers. Not decimals, not floats. Integers.
That’s a massive signal. Neural networks trained by gradient descent produce weights with many decimal places. Integer weights mean someone designed this network by hand, with a specific purpose. This wasn’t a learned model. It was a program in disguise.
This is the first senior debugging pattern: observe the shape of data before interpreting its content. A log with timestamps perfectly spaced every 100ms isn’t real traffic. A JSON where every field has exactly 3 elements didn’t come from production. The shape reveals the origin.
Phase 2: Reduce the problem space
Alex attempted to convert the network into a satisfiability (SAT) problem. The idea: if each neuron is a logical constraint, maybe a SAT solver could find inputs that produce output 1.
The reduction process was methodical:
- Eliminated 80% of neurons that were identity operations
- Merged nodes with a single input of weight 1
- Collapsed nodes with identical input vectors
From 2 million nodes down to 75,000. From there to 200,000 SAT variables.
It didn’t work. The problem remained intractable by brute force.
There’s a fundamental lesson here: correctly reducing a problem that still won’t solve is information, not failure. If after eliminating all accidental complexity the problem remains hard, the difficulty is inherent. That tells you something about the system’s nature. In this case, it suggested the function was probably irreversible — you couldn’t deduce the input from the output.
Phase 3: Pattern recognition
Alex noticed the network had 32 computational blocks that repeated periodically. Thirty-two identical rounds. An irreversible function.
He asked ChatGPT: “What cryptographic algorithms use 32 rounds?”
MD5.
That’s the eureka moment. But notice what made it possible: not a random intuition. It was the accumulation of three prior observations (integer weights → manual design, irreversibility → cryptography, 32 rounds → specific protocol) converging into a testable hypothesis.
This pattern — accumulating constraints until the possibility space collapses — is exactly how a senior diagnoses a production bug. It’s not that they know the answer beforehand. It’s that each observation eliminates entire categories of possibilities until only one remains.
Phase 4: The bug
Here the story gets interesting. Alex verified his hypothesis by computing MD5 of various inputs and comparing them to the network’s output. For inputs up to 32 characters, they matched perfectly.
For longer inputs, they didn’t.
The network had a bug. The designers had made an error in how they encoded message length — an overflow in the initial 7 layers that diverged from the MD5 standard for long inputs.
Alex traced the error layer by layer until he found the exact divergence.
This is textbook: verify the hypothesis at the edges. If your mental model says “this is MD5,” it’s not enough that it works for the happy path. You need to test it with long inputs, empty ones, special characters. And when it fails, the divergence tells you exactly where the error is.
Phase 5: Solve
With the function identified (MD5 with a known bug), Alex extracted the target hash from the second-to-last layer’s biases. Then he brute-forced with a dictionary: the answer was two English words separated by a space.
What this reveals about debugging
Alex’s process has nothing to do with machine learning. It’s pure debugging:
| Phase | In the puzzle | In real debugging |
|---|---|---|
| Observe shape | Integer weights → manual design | Uniform logs → synthetic data |
| Reduce problem | 2M nodes → 75K → SAT | Full stack trace → isolated component |
| Accumulate constraints | Irreversible + 32 rounds → cryptography | Fails only in prod + only Mondays → cron job |
| Verify at edges | Works <32 chars, fails >32 → overflow bug | Works with ASCII, fails UTF-8 → encoding |
| Use failure as clue | Overflow pinpoints exact layers | Stack trace points to exact line |
The structure is identical. The domain changes, but the reasoning is the same.
Meta-reasoning as competitive advantage
There’s a detail worth attention: Alex used ChatGPT for phase 3. Not to solve the puzzle, but to expand his search space. He had the constraints (irreversible, 32 rounds). He needed a catalog of candidates that met those constraints.
This is significant. The tool didn’t do the hard intellectual work — the observation, reduction, and constraint formulation phases were entirely human. What the tool did was act as external associative memory: “given what I know, what fits?”
This pattern will define senior work in the coming years. The valuable part isn’t knowing that MD5 has 32 rounds. Anyone can look that up. The valuable part is knowing you should search for “32-round cryptographic algorithm” and not “2,500-layer neural network.” Formulating the right question is the skill that doesn’t automate.
Interpretability: debugging the future
Jane Street’s puzzle is an exercise in mechanistic interpretability — a discipline that tries to understand what neural networks compute, not just evaluate whether their results are correct.
Today it’s a research field. In a few years it will be an operational necessity.
As more critical systems incorporate ML models — from medical diagnosis to financial decisions — the question “why did the model give this result?” stops being academic. European regulators already require it (AI Act). Product teams need it to debug false positives. Legal teams need it to respond to claims.
The professional profile emerging from this is interesting: someone who combines systems engineer thinking (layers, reduction, component isolation) with ML knowledge (architectures, activation functions, internal representations). Not a pure ML researcher. A model debugger.
The translation: if you know how to diagnose why a distributed system fails on Mondays between 3:00 and 3:15, you have 80% of the skills needed to diagnose why a model misclassifies images with blue backgrounds. The mental discipline is the same. The tools are different.
Implications for a senior today
The temptation when seeing such a puzzle is to think “this is for ML researchers, not me.” But the skills that solve it are the same ones that differentiate a senior from a mid-level in any domain:
1. Knowing what to look at. Alex didn’t try running the network millions of times to map inputs and outputs. He looked at the weights. A senior doesn’t look for the bug in yesterday’s code changes — they check logs, infrastructure, context.
2. Knowing when to change strategy. The SAT approach didn’t work. Alex didn’t insist. He switched to pattern recognition. A senior who’s spent 2 hours on a hypothesis that isn’t advancing discards it and seeks another.
3. Knowing what to ask. “What algorithm has 32 rounds and is irreversible?” is a precise question that produces a useful answer. “Why doesn’t my network work?” isn’t. Question quality determines answer quality — with LLMs and with human colleagues.
4. Using failures as information. The overflow bug wasn’t an obstacle. It was proof that the hypothesis was correct (MD5) and the clue to locate the divergence. In production, an intermittent 500 error isn’t a problem — it’s a symptom that leads you to the real problem.
Conclusion
Jane Street designed this puzzle as a recruiting tool. They’re looking for people who combine disciplines, who think in layers, who know when brute force doesn’t work and you need to change approach.
But what it really demonstrates is something broader: debugging skills are transferable across domains. Anyone who knows how to diagnose an opaque system — pulling the thread, accumulating constraints, verifying at the edges — can diagnose any opaque system. Whether it’s a data pipeline, a Kubernetes cluster, or a 2,500-layer neural network.
The question for any senior isn’t whether you need to learn ML. It’s whether your debugging method is rigorous enough to work when you change domains. Because domains change. The method, if it’s good, doesn’t.
The original puzzle is available on Hugging Face, and there’s a second open challenge where the network layers are scrambled and need reordering. If you’re up for the challenge, Jane Street’s complete article details the solution process.