On June 26 we released MirrorCode, a new AI benchmark made up of very long-horizon coding tasks. MirrorCode asks: What’s the largest software project AI can complete on its own?

This was a major project for me. I’m pleased that MirrorCode is out and seems to be gathering some attention, e.g. from Jack Clark in his Import AI newsletter.

The paper is very serious and buttoned-down; in this post I want to share some personal reflections and hot takes.

AI is progressing so quickly that the key tension in evals is now: how to make tasks that are difficult for AI, but actually fair? Since developing MirrorCode, my default expectation when I see a coding eval with low scores is that the tasks turn out to be impossible, models were grievously under-elicited, or both.¹

In MirrorCode, we leverage reimplementation of existing software as the raw material to make tasks hard. But we also manually select each of the 25 target programs, and carefully collect the end-to-end tests, so that the task is actually achievable (if extremely difficult).

If you’re not familiar with it, here’s a quick summary of the MirrorCode setup: AIs must reimplement a target program without access to its source code or the internet. They’re given the target’s docs, and a reference binary to which they can send inputs. The AI’s implementation must match the reference stdout and stderr exactly, for a long list of test inputs.

Avoiding absurd reverse-engineering

It’s very easy for MirrorCode-style tasks to degenerate into absurd reverse-engineering: stuff you’d never do as a human engineer. For example, one of our target programs is the brotli compression library. But we explicitly test the decompression path only (and call the task “brotlid”).

Brotli’s RFC 7932 only defines decompression: given compressed bytes, there’s one correct output. In the other direction, there’s no unique mapping. Given input text, there are many valid compressions of it, some more efficient than others.

So the output of a compressor is governed by thousands of internal efficiency heuristics (e.g. how far back to look for repeated strings). Replicating those perfectly from a black box is extreme and pointless reverse-engineering.

We put considerable effort into avoiding this by selecting appropriate target programs and curating test inputs. This requires judgement. Occasionally it can be quite subjective where to draw the line for “extreme” reverse-engineering, but in most cases it’s not close (like in the brotli example).

Telling AIs what’s in scope

Another key differentiator of MirrorCode: AIs are clearly told the scope of the task, rather than having to guess what they’ll secretly be tested on. We clarify task scope by showing AIs a list of test inputs (while also keeping some hidden to prevent AIs from cheating with a lookup table). If this sounds too easy or like giving them the answer, I assure you, you’re thinking about it wrong. What we do is completely different from showing the answer in a Q&A benchmark (Q&A is where the wrong intuition comes from, IMO). This is more like defining what question you’re even asking.

For example, perfectly matching gotree’s behaviour does not become easy because I show you 1,899 inputs (!) that your program will be tested on.

Personally I think it would take me >1 month without AI and using the stdlib only (the setting we test AIs in).

Meanwhile, if you don’t show test cases and just say “reimplement all of gotree” the task is dominated by an impossible guessing-game: what’s actually in scope?

It’s obviously infeasible to search across literally all of the possible input strings that gotree could be run on. Documentation helps to narrow this down, but cannot cover all of the complex ways that the software is used in reality.

As just one of many examples of what happens when you look at real-world trees: comments in trees.

The Nexus format was only loosely specified in its original publication (which we give to AIs). Enumerating every real-world use of the Nexus format, from scratch (no looking up online!), is not feasible. Among other omissions, documentation does not mention that Nexus files may contain comments. Nexus comments are in brackets (e.g. [comment]), so they may appear in the middle of a line of data, unlike code comments. Real-world files produced by standard tools contain comments in various formats (e.g. multiline comments). gotree generally handles them without complaint, but errors with comments in certain locations (comments inside the TAXLABELS clause if you need to know).

I personally think guessing that comments exist, and all the ways comments must be handled, despite comments not even being mentioned in documentation, is functionally impossible. But everyone should at least agree that guessing this is far harder than actually implementing comment-handling. So the benchmark becomes dominated by the guessing-game, rather than actual implementation.

Human SWEs gradually learn the scope of inputs a program should support thanks to external feedback from users of the program, they don’t just think them all up in a vacuum before publication. Yet that’s what some benchmarks test.

btw, don’t underestimate these tasks

At least if Hacker News commenters are anything to go by, I think it’s easy to underestimate some of our target programs. Take the calendar utility cal. It seems simple: print a calendar, trivial right?

But then you think about what’s actually involved in a fully accurate calendar with a lot of configurable options, and the need for byte-exact output on 1,365 test cases…

For example, you need to think about September 1752, which is the date the British switched from the Julian to the Gregorian calendar, dropping 11 days. Of course.

We use cal from util-linux. For example, a user can pass 11 2026 to print a calendar for November 2026. By default, the first day of the week is determined by the system’s locale, but it can be explicitly overridden with --monday or --sunday.

cal supports numerous features:

1. Layouts. By default cal displays one month. The flag --months displays an arbitrary number of months in a grid layout, whose width is controlled by --columns. --week additionally prints week numbers. The --vertical flag flips the layout, making each week a column instead of a row, and causes week numbers to appear at the bottom instead of the left. -j uses day-of-year numbering instead of day-of-month; the layout must then expand to accommodate three-digit numbers. To pass the task, an AI must produce byte-exact output for all these layouts.
2. Choice of calendar system. By default, the Gregorian calendar is used from 3 September 1752, and the Julian before that. 11 days were removed at the time of adoption to bring the calendar in sync with solar events, so September 1752 has a mix of Julian and Gregorian dates by which the 2nd is followed by the 14th (the 3rd through the 13th are absent). The --reform flag customizes the date of adoption of the Gregorian calendar. Separately, week numbering depends on the choice of first day of the week: --sunday implies 1 January is in week 1, while --monday implies the ISO8601 standard, where the first Thursday is in week 1.
3. Date input formats. Numerous input formats are supported, such as 2020, 11 2020, 25-11-2020, August, Aug, yesterday, tomorrow, +3months.

Would you back yourself to implement all this (without AI), passing 100% of tests, in under a week? I’m not sure I would.

Security mindset for evals

I’m especially proud of this one. MirrorCode is the first benchmark I know of to implement a serious “security mindset”. This means treating AI cheating not as an afterthought, but as a first-class consideration in every aspect of eval design.

Let’s look at three examples from other benchmarks. With SWE-bench,² KernelBench,³ and RE-bench,⁴ brittle defences were bolted on after someone caught AIs cheating in a specific way. The KernelBench maintainers concede that “even when we patch known exploits, new ones emerge, often even harder to detect”, and the RE-bench authors lamented having to resort to laborious manual inspection.

I get the sense that the field fatalistically treats AI cheating as an unwinnable game of whack-a-mole. I think this is completely wrong. Most benchmarks can be made secure, by using basic primitives like containers and privilege separation. We just need to take this seriously and actually do the obvious things. Most authors of new evals are behind the curve. They aren’t even trying to make their evals secure against human-level attackers, which AIs are. That’s why cheating comes up all the time, not because it’s inherently that hard to prevent.

See Appendix D of the paper for the details of how I did it in MirrorCode.

It’s open source

MirrorCode is open source, and it already supports lots of variations: e.g. ablate away test case visibility, show agents the source code, disable the mechanisms that force agents to keep working even when they want to stop, etc.

I have about a thousand research ideas for things that would be easy to try on top of this harness. I won’t have time to do most of them. Would love it if people just tried stuff! It’s free real estate. For example, prompting changes would likely help certain models improve their scores; our prompt just describes the task in a matter-of-fact way.⁵ Or: can you come up with a target program that the best models profoundly struggle with, getting very few test cases to pass at all? As of Claude Fable 5, none of our tasks meet this bar; I think it’s actually a very difficult challenge and would be impressed if someone solved it.

A few inference scaling curves

MirrorCode lets us measure partial progress (% of test cases passed) during an attempt, which leads to nice inference scaling curves. I hope to study this more systematically in the future, but for now here’s a cherry-picked collection of interesting curves.

Older models struggling (GPT-5 and Opus 4.1, both from August 2025):