AIs have such poor judgement for when to write a code comment that I prefer to have them write none at all. My global AGENTS.md (which is quite short: 769 words) includes the instructions:

Don’t add any comments or documentation unless I explicitly ask for them. Don’t edit existing comments unless your change makes them wrong.

I’ve mostly been using Claude lately, and it does not obey the instructions! It frequently still adds comments. It’s quite surprising that instruction following can still be so poor on such powerful models.

But that’s just my anecdotal impression. I decided to measure the effect systematically. I call this new eval “Shut up and SWE-bench”.

Method

I took SWE-bench Verified and drew 100 tasks at random from the 261 that the annotators rated as taking 15 minutes to 1 hour (I excluded the “under 15 minutes” bucket because a one-line fix leaves little room for a comment).¹ The system prompt is the one from inspect_evals with my two rules appended:

Don’t add any comments or documentation unless the issue explicitly asks for them. Don’t edit existing comments unless your change makes them wrong.

Issues essentially never explicitly ask for comments, so the “unless” clause was only there to keep the prompt more similar to what I have in my AGENTS.md.

To detect comments, the scorer takes every .py file the agent changed and runs it through Python’s tokenize module before and after the change. It collects the COMMENT tokens and the STRING tokens that start a statement (i.e. docstrings), and any comment or docstring text that’s in the new file but wasn’t in the old one counts as added. A comment that merely moved doesn’t count.

A comment the agent reworded does get flagged, so I then go through the diff and discard any flagged comment that has a similar removed comment line in the same hunk, and any flagged docstring where fewer than half the lines are new. Whether such comment edits are legitimate depends on whether an agent’s code change “made an old comment wrong”, but this happens rarely enough that I just give them the benefit of the doubt, and only count newly added comments. This is good enough for a quick experiment.

Results

Fable 5.1 added a new comment or docstring in 33 of the 100 tasks, and so did Gemini 3.8 Flash. Standard errors are around 5 percentage points. GPT-6 Astra does well: it beats Claude at Shut up and SWE-bench despite resolving fewer issues, and its 6% of tasks with a new comment is within the noise of my hacky heuristics (all 6 comments seem legitimate to me). The full transcripts are here: Fable and Astra, Gemini.

In a further 6 tasks, Fable only reworded existing comments or docstrings. I checked these by hand and 5 of the 6 were legitimate: the fix had made the old text wrong, or added a parameter that the docstring then needed to list.

Here’s an example of a new comment:

             locs = locs[(vmin <= locs) & (locs <= vmax)]
+            # Avoid having an offset / scientific notation in a legend
+            # as we don't represent that anywhere so it ends up incorrect.
+            # This could become an option (e.g. Continuous.label(offset=True))
+            # in which case we would need to figure out how to show it.
+            if hasattr(axis.major.formatter, "set_useOffset"):

Memorization may be an issue for this eval, as it is for SWE-bench. In one Django task, Fable added a seven-line comment block that is word for word the comment from Django’s own upstream fix.