AIs have such poor judgement for when to write a code comment that I prefer to have them write none at all. My global AGENTS.md (which is quite short: 769 words) includes the instructions:

Don’t add any comments or documentation unless I explicitly ask for them. Don’t edit existing comments unless your change makes them wrong.

I’ve mostly been using Claude lately, and it does not obey the instructions! It frequently still adds comments. It’s quite surprising that instruction following can still be so poor on such powerful models.

But that’s just my anecdotal impression. I decided to measure the effect systematically. I call this new eval “Shut up and SWE-bench”.

Method

I took SWE-bench Verified and drew 100 tasks at random from the 261 that the annotators rated as taking 15 minutes to 1 hour (I excluded the “under 15 minutes” bucket because a one-line fix leaves little room for a comment).¹ The system prompt is the one from inspect_evals with my two rules appended:

Don’t add any comments or documentation unless the issue explicitly asks for them. Don’t edit existing comments unless your change makes them wrong.

Issues essentially never explicitly ask for comments, so the “unless” clause was only there to keep the prompt more similar to what I have in my AGENTS.md.

To detect comments, the scorer takes every .py file the agent changed and runs it through Python’s tokenize module before and after the change. It collects the COMMENT tokens and the STRING tokens that start a statement (i.e. docstrings), and any comment or docstring text that’s in the new file but wasn’t in the old one counts as added. A comment that merely moved doesn’t count.

A comment the agent reworded does get flagged, so I then go through the diff and discard any flagged comment that has a similar removed comment line in the same hunk, and any flagged docstring where fewer than half the lines are new. Whether such comment edits are legitimate depends on whether an agent’s code change “made an old comment wrong”, but this happens rarely enough that I just give them the benefit of the doubt, and only count newly added comments. This is good enough for a quick experiment.

Results

Fable 5.1 added a new comment or docstring in 33 of the 100 tasks, and so did Gemini 3.8 Flash. Standard errors are around 5 percentage points. GPT-6 Astra does well: it beats Claude at Shut up and SWE-bench despite resolving fewer issues, and its 6% of tasks with a new comment is within the noise of my hacky heuristics (all 6 comments seem legitimate to me). The full transcripts are here: Fable and Astra, Gemini.

In a further 6 tasks, Fable only reworded existing comments or docstrings. I checked these by hand and 5 of the 6 were legitimate: the fix had made the old text wrong, or added a parameter that the docstring then needed to list.

Here’s an example of a new comment:

             locs = locs[(vmin <= locs) & (locs <= vmax)]
+            # Avoid having an offset / scientific notation in a legend
+            # as we don't represent that anywhere so it ends up incorrect.
+            # This could become an option (e.g. Continuous.label(offset=True))
+            # in which case we would need to figure out how to show it.
+            if hasattr(axis.major.formatter, "set_useOffset"):

Memorization may be an issue for this eval, as it is for SWE-bench. In one Django task, Fable added a seven-line comment block that is word for word the comment from Django’s own upstream fix.

On June 26 we released MirrorCode, a new AI benchmark made up of very long-horizon coding tasks. MirrorCode asks: What’s the largest software project AI can complete on its own?

This was a major project for me. I’m pleased that MirrorCode is out and seems to be gathering some attention, e.g. from Jack Clark in his Import AI newsletter.

The paper is very serious and buttoned-down; in this post I want to share some personal reflections and hot takes.

AI is progressing so quickly that the key tension in evals is now: how to make tasks that are difficult for AI, but actually fair? Since developing MirrorCode, my default expectation when I see a coding eval with low scores is that the tasks turn out to be impossible, models were grievously under-elicited, or both.¹

In MirrorCode, we leverage reimplementation of existing software as the raw material to make tasks hard. But we also manually select each of the 25 target programs, and carefully collect the end-to-end tests, so that the task is actually achievable (if extremely difficult).

If you’re not familiar with it, here’s a quick summary of the MirrorCode setup: AIs must reimplement a target program without access to its source code or the internet. They’re given the target’s docs, and a reference binary to which they can send inputs. The AI’s implementation must match the reference stdout and stderr exactly, for a long list of test inputs.

Avoiding absurd reverse-engineering

It’s very easy for MirrorCode-style tasks to degenerate into absurd reverse-engineering: stuff you’d never do as a human engineer. For example, one of our target programs is the brotli compression library. But we explicitly test the decompression path only (and call the task “brotlid”).

Brotli’s RFC 7932 only defines decompression: given compressed bytes, there’s one correct output. In the other direction, there’s no unique mapping. Given input text, there are many valid compressions of it, some more efficient than others.

So the output of a compressor is governed by thousands of internal efficiency heuristics (e.g. how far back to look for repeated strings). Replicating those perfectly from a black box is extreme and pointless reverse-engineering.

We put considerable effort into avoiding this by selecting appropriate target programs and curating test inputs. This requires judgement. Occasionally it can be quite subjective where to draw the line for “extreme” reverse-engineering, but in most cases it’s not close (like in the brotli example).

Telling AIs what’s in scope

Another key differentiator of MirrorCode: AIs are clearly told the scope of the task, rather than having to guess what they’ll secretly be tested on. We clarify task scope by showing AIs a list of test inputs (while also keeping some hidden to prevent AIs from cheating with a lookup table). If this sounds too easy or like giving them the answer, I assure you, you’re thinking about it wrong. What we do is completely different from showing the answer in a Q&A benchmark (Q&A is where the wrong intuition comes from, IMO). This is more like defining what question you’re even asking.

For example, perfectly matching gotree’s behaviour does not become easy because I show you 1,899 inputs (!) that your program will be tested on.

Personally I think it would take me >1 month without AI and using the stdlib only (the setting we test AIs in).

Meanwhile, if you don’t show test cases and just say “reimplement all of gotree” the task is dominated by an impossible guessing-game: what’s actually in scope?

It’s obviously infeasible to search across literally all of the possible input strings that gotree could be run on. Documentation helps to narrow this down, but cannot cover all of the complex ways that the software is used in reality.

As just one of many examples of what happens when you look at real-world trees: comments in trees.

The Nexus format was only loosely specified in its original publication (which we give to AIs). Enumerating every real-world use of the Nexus format, from scratch (no looking up online!), is not feasible. Among other omissions, documentation does not mention that Nexus files may contain comments. Nexus comments are in brackets (e.g. [comment]), so they may appear in the middle of a line of data, unlike code comments. Real-world files produced by standard tools contain comments in various formats (e.g. multiline comments). gotree generally handles them without complaint, but errors with comments in certain locations (comments inside the TAXLABELS clause if you need to know).

I personally think guessing that comments exist, and all the ways comments must be handled, despite comments not even being mentioned in documentation, is functionally impossible. But everyone should at least agree that guessing this is far harder than actually implementing comment-handling. So the benchmark becomes dominated by the guessing-game, rather than actual implementation.

Human SWEs gradually learn the scope of inputs a program should support thanks to external feedback from users of the program, they don’t just think them all up in a vacuum before publication. Yet that’s what some benchmarks test.

btw, don’t underestimate these tasks

At least if Hacker News commenters are anything to go by, I think it’s easy to underestimate some of our target programs. Take the calendar utility cal. It seems simple: print a calendar, trivial right?

But then you think about what’s actually involved in a fully accurate calendar with a lot of configurable options, and the need for byte-exact output on 1,365 test cases…

For example, you need to think about September 1752, which is the date the British switched from the Julian to the Gregorian calendar, dropping 11 days. Of course.

We use cal from util-linux. For example, a user can pass 11 2026 to print a calendar for November 2026. By default, the first day of the week is determined by the system’s locale, but it can be explicitly overridden with --monday or --sunday.

cal supports numerous features:

1. Layouts. By default cal displays one month. The flag --months displays an arbitrary number of months in a grid layout, whose width is controlled by --columns. --week additionally prints week numbers. The --vertical flag flips the layout, making each week a column instead of a row, and causes week numbers to appear at the bottom instead of the left. -j uses day-of-year numbering instead of day-of-month; the layout must then expand to accommodate three-digit numbers. To pass the task, an AI must produce exact output for all these layouts.
2. Choice of calendar system. By default, the Gregorian calendar is used from 3 September 1752, and the Julian before that. 11 days were removed at the time of adoption to bring the calendar in sync with solar events, so September 1752 has a mix of Julian and Gregorian dates by which the 2nd is followed by the 14th (the 3rd through the 13th are absent). The --reform flag customizes the date of adoption of the Gregorian calendar. Separately, week numbering depends on the choice of first day of the week: --sunday implies 1 January is in week 1, while --monday implies the ISO8601 standard, where the first Thursday is in week 1.
3. Date input formats. Numerous input formats are supported, such as 2020, 11 2020, 25-11-2020, August, Aug, yesterday, tomorrow, +3months.

Would you back yourself to implement all this (without AI), passing 100% of tests, in under a week? I’m not sure I would.

Security mindset for evals

I’m especially proud of this one. MirrorCode is the first benchmark I know of to implement a serious “security mindset”. This means treating AI cheating not as an afterthought, but as a first-class consideration in every aspect of eval design.

Let’s look at three examples from other benchmarks. With SWE-bench,² KernelBench,³ and RE-bench,⁴ brittle defences were bolted on after someone caught AIs cheating in a specific way. The KernelBench maintainers concede that “even when we patch known exploits, new ones emerge, often even harder to detect”, and the RE-bench authors lamented having to resort to laborious manual inspection.

I get the sense that the field fatalistically treats AI cheating as an unwinnable game of whack-a-mole. I think this is completely wrong. Most benchmarks can be made secure, by using basic primitives like containers and privilege separation. We just need to take this seriously and actually do the obvious things. Most authors of new evals are behind the curve. They aren’t even trying to make their evals secure against human-level attackers, which AIs are. That’s why cheating comes up all the time, not because it’s inherently that hard to prevent.

See Appendix D of the paper for the details of how I did it in MirrorCode.

It’s open source

MirrorCode is open source, and it already supports lots of variations: e.g. ablate away test case visibility, show agents the source code, disable the mechanisms that force agents to keep working even when they want to stop, etc.

I have about a thousand research ideas for things that would be easy to try on top of this harness. I won’t have time to do most of them. Would love it if people just tried stuff! It’s free real estate. For example, prompting changes would likely help certain models improve their scores; our prompt just describes the task in a matter-of-fact way.⁵ Or: can you come up with a target program that the best models profoundly struggle with, getting very few test cases to pass at all? As of Claude Fable 5, none of our tasks meet this bar; I think it’s actually a very difficult challenge and would be impressed if someone solved it.

A few inference scaling curves

MirrorCode lets us measure partial progress (% of test cases passed) during an attempt, which leads to nice inference scaling curves. I hope to study this more systematically in the future, but for now here’s a cherry-picked collection of interesting curves.

Older models struggling (GPT-5 and Opus 4.1, both from August 2025):

In a previous post about creating a Docker registry of SWE-bench images, I made a brief side note about hypothetical reward hacking:

[…] as a side note, it’s worth asking whether the git history should be included at all. It makes sense for the model to have access to the past git history, like a human developer would. However, the model should not have access to future history from after the PR was merged. A sophisticated cheating model could in theory reward hack the evaluation if there is any way to access future history. I believe that is possible in some circumstances even after a git reset --hard and git remote remove origin. For example, if version tags are used, the model could git checkout <tag> a future tag. Or it could access the dangling commits in some way, perhaps with git fsck --lost-found. I believe this is unlikely to be a problem currently, but could become one with future models.

Less than two months later, it seems that this behaviour has now been seen in the wild!

A META AI researcher has just posted an issue on the SWE-bench repo:

We’ve identified multiple loopholes with SWE Bench Verified where agents may look at future repository state (by querying it directly or through a variety of methods), and cases in which future repository state includes either solutions or detailed approaches to solving problems (commit messages and more).

… In a trajectory with Claude 4 Sonnet, pytest-dev__pytest-6202, the agent uses git log --all which leaks future commits that directly fix the issue.

Claude 4 Sonnet came out in May 2025, and I wrote my post in July. I was quite wrong about how long it would take for models to become sophisticated enough to cheat in this way. They were already doing it. I had never seen a model attempt this loophole, but one man can only read so many SWE-bench traces.

The command git log --all (which I didn’t know about) shows all commits reachable from any ref. A ref is any reference to the SHA-1 value of a commit, found under .git/refs, and includes: (1) the heads of branches, (2) tags, (3) remotes, and (4) any manually created refs. As I explained in the previous post, the original SWE-bench authors only removed the default remote, with a code comment: “Remove the remote so the agent won’t see newer commits”. But they didn’t think about other refs, like tags (a fix is underway).

git log --all lets a cheating model efficiently find any (non-dangling) commits, without having to look through the refs and figure out which one might point to future history and be usable for cheating. For example, Claude 4 Sonnet cleverly searched for commit messages with keywords relevant to the issue: git log --oneline --all | grep -i "bracket|parametrize|modpath".

In the pytest-6202 example, the issue is from November 2019. git log --all shows commits up to d0f136f (September 2024), which is tagged as 8.3.3. Indeed 8.3.3 seems to be the highest version tag present on the main branch:

$ git tag --list
1.0.0
1.0.0b3
1.0.0b6
...many more tags...
8.3.2
8.3.3
8.4.0.dev0
v7.4.3

In the example issues I looked at, git tags were the reason the future history was available. This is exactly the vulnerability that had jumped out at me when I looked into how the Docker images are constructed.

In pytest-6202, deleting all tags is sufficient to remove the offending commits from git log --all. However, a git fsck --lost-found will uncover the dangling tags and commits (ones not pointed to by any ref).

Here we must get into one subtlety. git show-ref 8.3.3 will show d430e32, not d0f136f, the actual September 2024 commit. d430e32 is the SHA-1 of the annotated tag object, which itself points to the actual commit d0f136f. git show-ref --dereference 8.3.3 can be used to show d0f136f.

Even after deleting tags, git fsck --lost-found will uncover

$ git fsck --lost-found
...
dangling tag d430e325c6d8c7161ae2e468ea5045a163e4c517
...

A cheater can then directly do git checkout d430e32, which moves the HEAD to the September 2024 commit d0f136f. Since HEAD is a ref, git log --all will at that point show the entire future history, just like before we deleted the tags (in fact you don’t need the --all now).

My understanding is that git gc --prune=now (docs) removes any unreachable objects. The documentation says:

git gc tries very hard not to delete objects that are referenced anywhere in your repository. In particular, it will keep not only objects referenced by your current set of branches and tags, but also objects referenced by the index, remote-tracking branches, reflogs (which may reference commits in branches that were later amended or rewound), and anything else in the refs/* namespace.

The reflog and index are strictly local, so there should be nothing in them just after cloning. We’ve covered branches, tags, and remotes. The only exception I can think of are manually created refs (using git update-ref), which should be extremely unusual. To mitigate this, one could trust only the HEAD ref and delete all others.

I spent a couple of hours today trying to find other loopholes that work even after git gc. I scoured various git internals I had never thought about before, like packfiles (in .git/objects/pack), or .git/objects/info/commit-graph. None of these worked. But git is a big program with a lot of arcane features, and I only tried a couple of SWE-bench issues. Can you find a more sophisticated hack?