bayes.net

Shut up and SWE-bench

2026-09-24T00:00:00+00:00

AIs have such poor judgement for when to write a code comment that I prefer to have them write none at all. My global AGENTS.md (which is quite short: 769 words) includes the instructions:

Don’t add any comments or documentation unless I explicitly ask for them. Don’t edit existing comments unless your change makes them wrong.

I’ve mostly been using Claude lately, and it does not obey the instructions! It frequently still adds comments. It’s quite surprising that instruction following can still be so poor on such powerful models.

But that’s just my anecdotal impression. I decided to measure the effect systematically. I call this new eval “Shut up and SWE-bench”.

Method

I took SWE-bench Verified and drew 100 tasks at random from the 261 that the annotators rated as taking 15 minutes to 1 hour (I excluded the “under 15 minutes” bucket because a one-line fix leaves little room for a comment).¹ The system prompt is the one from inspect_evals with my two rules appended:

Don’t add any comments or documentation unless the issue explicitly asks for them. Don’t edit existing comments unless your change makes them wrong.

Issues essentially never explicitly ask for comments, so the “unless” clause was only there to keep the prompt more similar to what I have in my AGENTS.md.

To detect comments, the scorer takes every .py file the agent changed and runs it through Python’s tokenize module before and after the change. It collects the COMMENT tokens and the STRING tokens that start a statement (i.e. docstrings), and any comment or docstring text that’s in the new file but wasn’t in the old one counts as added. A comment that merely moved doesn’t count.

A comment the agent reworded does get flagged, so I then go through the diff and discard any flagged comment that has a similar removed comment line in the same hunk, and any flagged docstring where fewer than half the lines are new. Whether such comment edits are legitimate depends on whether an agent’s code change “made an old comment wrong”, but this happens rarely enough that I just give them the benefit of the doubt, and only count newly added comments. This is good enough for a quick experiment.

Results

Model	Resolved	Added no comments	Both (Shut up and SWE-bench score)
Claude Fable 5.1	86%	67%	59%
GPT-6 Astra	80%	94%	76%
Gemini 3.8 Flash	76%	67%	55%

Fable 5.1 added a new comment or docstring in 33 of the 100 tasks, and so did Gemini 3.8 Flash. Standard errors are around 5 percentage points. GPT-6 Astra does well: it beats Claude at Shut up and SWE-bench despite resolving fewer issues, and its 6% of tasks with a new comment is within the noise of my hacky heuristics (all 6 comments seem legitimate to me). The full transcripts are here: Fable and Astra, Gemini.

In a further 6 tasks, Fable only reworded existing comments or docstrings. I checked these by hand and 5 of the 6 were legitimate: the fix had made the old text wrong, or added a parameter that the docstring then needed to list.

Here’s an example of a new comment:

             locs = locs[(vmin <= locs) & (locs <= vmax)]
+            # Avoid having an offset / scientific notation in a legend
+            # as we don't represent that anywhere so it ends up incorrect.
+            # This could become an option (e.g. Continuous.label(offset=True))
+            # in which case we would need to figure out how to show it.
+            if hasattr(axis.major.formatter, "set_useOffset"):

Memorization may be an issue for this eval, as it is for SWE-bench. In one Django task, Fable added a seven-line comment block that is word for word the comment from Django’s own upstream fix.

It was nice to discover that inspect_evals uses the optimized SWE-bench image registry I built for Epoch AI in July 2025 as its default. It still works: the images pulled in about 20 seconds each, and 100 agent runs plus test suites took 14 minutes on our cluster. ↩

MirrorCode and making evals that are hard, but actually fair

2026-07-30T00:00:00+00:00

On June 26 we released MirrorCode, a new AI benchmark made up of very long-horizon coding tasks. MirrorCode asks: What’s the largest software project AI can complete on its own?

This was a major project for me. I’m pleased that MirrorCode is out and seems to be gathering some attention, e.g. from Jack Clark in his Import AI newsletter.

The paper is very serious and buttoned-down; in this post I want to share some personal reflections and hot takes.

AI is progressing so quickly that the key tension in evals is now: how to make tasks that are difficult for AI, but actually fair? Since developing MirrorCode, my default expectation when I see a coding eval with low scores is that the tasks turn out to be impossible, models were grievously under-elicited, or both.¹

In MirrorCode, we leverage reimplementation of existing software as the raw material to make tasks hard. But we also manually select each of the 25 target programs, and carefully collect the end-to-end tests, so that the task is actually achievable (if extremely difficult).

If you’re not familiar with it, here’s a quick summary of the MirrorCode setup: AIs must reimplement a target program without access to its source code or the internet. They’re given the target’s docs, and a reference binary to which they can send inputs. The AI’s implementation must match the reference stdout and stderr exactly, for a long list of test inputs.

Avoiding absurd reverse-engineering

It’s very easy for MirrorCode-style tasks to degenerate into absurd reverse-engineering: stuff you’d never do as a human engineer. For example, one of our target programs is the brotli compression library. But we explicitly test the decompression path only (and call the task “brotlid”).

Brotli’s RFC 7932 only defines decompression: given compressed bytes, there’s one correct output. In the other direction, there’s no unique mapping. Given input text, there are many valid compressions of it, some more efficient than others.

So the output of a compressor is governed by thousands of internal efficiency heuristics (e.g. how far back to look for repeated strings). Replicating those perfectly from a black box is extreme and pointless reverse-engineering.

We put considerable effort into avoiding this by selecting appropriate target programs and curating test inputs. This requires judgement. Occasionally it can be quite subjective where to draw the line for “extreme” reverse-engineering, but in most cases it’s not close (like in the brotli example).

Telling AIs what’s in scope

Another key differentiator of MirrorCode: AIs are clearly told the scope of the task, rather than having to guess what they’ll secretly be tested on. We clarify task scope by showing AIs a list of test inputs (while also keeping some hidden to prevent AIs from cheating with a lookup table). If this sounds too easy or like giving them the answer, I assure you, you’re thinking about it wrong. What we do is completely different from showing the answer in a Q&A benchmark (Q&A is where the wrong intuition comes from, IMO). This is more like defining what question you’re even asking.

For example, perfectly matching gotree’s behaviour does not become easy because I show you 1,899 inputs (!) that your program will be tested on.

Personally I think it would take me >1 month without AI and using the stdlib only (the setting we test AIs in).

A slice of gotree’s test cases. Cases marked hidden are held out from the AI.

Meanwhile, if you don’t show test cases and just say “reimplement all of gotree” the task is dominated by an impossible guessing-game: what’s actually in scope?

It’s obviously infeasible to search across literally all of the possible input strings that gotree could be run on. Documentation helps to narrow this down, but cannot cover all of the complex ways that the software is used in reality.

As just one of many examples of what happens when you look at real-world trees: comments in trees.

The Nexus format was only loosely specified in its original publication (which we give to AIs). Enumerating every real-world use of the Nexus format, from scratch (no looking up online!), is not feasible. Among other omissions, documentation does not mention that Nexus files may contain comments. Nexus comments are in brackets (e.g. [comment]), so they may appear in the middle of a line of data, unlike code comments. Real-world files produced by standard tools contain comments in various formats (e.g. multiline comments). gotree generally handles them without complaint, but errors with comments in certain locations (comments inside the TAXLABELS clause if you need to know).

I personally think guessing that comments exist, and all the ways comments must be handled, despite comments not even being mentioned in documentation, is functionally impossible. But everyone should at least agree that guessing this is far harder than actually implementing comment-handling. So the benchmark becomes dominated by the guessing-game, rather than actual implementation.

Human SWEs gradually learn the scope of inputs a program should support thanks to external feedback from users of the program, they don’t just think them all up in a vacuum before publication. Yet that’s what some benchmarks test.

btw, don’t underestimate these tasks

At least if Hacker News commenters are anything to go by, I think it’s easy to underestimate some of our target programs. Take the calendar utility cal. It seems simple: print a calendar, trivial right?

But then you think about what’s actually involved in a fully accurate calendar with a lot of configurable options, and the need for byte-exact output on 1,365 test cases…

For example, you need to think about September 1752, which is the date the British switched from the Julian to the Gregorian calendar, dropping 11 days. Of course.

We use cal from util-linux. For example, a user can pass 11 2026 to print a calendar for November 2026. By default, the first day of the week is determined by the system’s locale, but it can be explicitly overridden with --monday or --sunday.

cal supports numerous features:

Layouts. By default cal displays one month. The flag --months displays an arbitrary number of months in a grid layout, whose width is controlled by --columns. --week additionally prints week numbers. The --vertical flag flips the layout, making each week a column instead of a row, and causes week numbers to appear at the bottom instead of the left. -j uses day-of-year numbering instead of day-of-month; the layout must then expand to accommodate three-digit numbers. To pass the task, an AI must produce exact output for all these layouts.
Choice of calendar system. By default, the Gregorian calendar is used from 3 September 1752, and the Julian before that. 11 days were removed at the time of adoption to bring the calendar in sync with solar events, so September 1752 has a mix of Julian and Gregorian dates by which the 2nd is followed by the 14th (the 3rd through the 13th are absent). The --reform flag customizes the date of adoption of the Gregorian calendar. Separately, week numbering depends on the choice of first day of the week: --sunday implies 1 January is in week 1, while --monday implies the ISO8601 standard, where the first Thursday is in week 1.
Date input formats. Numerous input formats are supported, such as 2020, 11 2020, 25-11-2020, August, Aug, yesterday, tomorrow, +3months.

Would you back yourself to implement all this (without AI), passing 100% of tests, in under a week? I’m not sure I would.

Representative cal invocations and their output, illustrating the default, --monday, --week, vertical multi-month, and day-of-year (-j) layouts.

Security mindset for evals

I’m especially proud of this one. MirrorCode is the first benchmark I know of to implement a serious “security mindset”. This means treating AI cheating not as an afterthought, but as a first-class consideration in every aspect of eval design.

Let’s look at three examples from other benchmarks. With SWE-bench,² KernelBench,³ and RE-bench,⁴ brittle defences were bolted on after someone caught AIs cheating in a specific way. The KernelBench maintainers concede that “even when we patch known exploits, new ones emerge, often even harder to detect”, and the RE-bench authors lamented having to resort to laborious manual inspection.

I get the sense that the field fatalistically treats AI cheating as an unwinnable game of whack-a-mole. I think this is completely wrong. Most benchmarks can be made secure, by using basic primitives like containers and privilege separation. We just need to take this seriously and actually do the obvious things. Most authors of new evals are behind the curve. They aren’t even trying to make their evals secure against human-level attackers, which AIs are. That’s why cheating comes up all the time, not because it’s inherently that hard to prevent.

See Appendix D of the paper for the details of how I did it in MirrorCode.

Figure from Appendix D: MirrorCode’s container architecture (one task instance).

It’s open source

MirrorCode is open source, and it already supports lots of variations: e.g. ablate away test case visibility, show agents the source code, disable the mechanisms that force agents to keep working even when they want to stop, etc.

I have about a thousand research ideas for things that would be easy to try on top of this harness. I won’t have time to do most of them. Would love it if people just tried stuff! It’s free real estate. For example, prompting changes would likely help certain models improve their scores; our prompt just describes the task in a matter-of-fact way.⁵ Or: can you come up with a target program that the best models profoundly struggle with, getting very few test cases to pass at all? As of Claude Fable 5, none of our tasks meet this bar; I think it’s actually a very difficult challenge and would be impressed if someone solved it.

A few inference scaling curves

MirrorCode lets us measure partial progress (% of test cases passed) during an attempt, which leads to nice inference scaling curves. I hope to study this more systematically in the future, but for now here’s a cherry-picked collection of interesting curves.

Slow grind on pkl.

High variance of token usage for texmacros.

Older models struggling (GPT-5 and Opus 4.1, both from August 2025):

This doesn’t mean there are no good evals out there. It’s only my prior, if the only thing I know about a coding benchmark is that scores are low. ↩
The SWE-bench evaluation environments are built from a clone of each task’s repository, whose git history extends past the task’s base commit — including the commit that fixed the issue in question. When the containerized, agent-facing setup was introduced in June 2024, it removed the origin remote “so the agent won’t see newer commits”. This proved insufficient: version tags and the reflog still pointed at post-base-commit history, and by mid-2025 agents such as Claude 4 Sonnet were observed running git log --all to recover the upstream fix commit and copy its patch (I wrote about this at the time). After the loopholes were reported in September 2025, the maintainers patched the setup again: deleting tags newer than the base commit, expiring the reflog, garbage-collecting unreachable objects, and asserting that no future commits remain visible. ↩
A KernelBench task gives the model a reference PyTorch module (e.g. a convolution layer) and asks it to write a custom GPU kernel that computes the same outputs faster; submissions are scored by comparing outputs against the reference on sampled inputs and timing the speedup. After the late-2024 release, models were found scoring well without writing real kernels: calling PyTorch’s own operators or cuBLAS instead of implementing them, writing a kernel but never invoking it, emitting no-ops that passed the correctness check because the evaluation read outputs from the same memory the reference had just written, and hiding computation from the timer on a separate CUDA stream. Countermeasures became a string-matching arms race. When Cognition used KernelBench tasks for RL training in 2025, they zeroed the reward of any solution containing the string torch.nn; the model responded by wrapping broken kernels in try/except blocks that fell back to PyTorch; they banned those keywords; it then subclassed the reference implementation with pass, which was banned in turn. The maintainers’ July 2025 v0.1 release rescaled problems and randomized test inputs. ↩
On METR’s RE-bench, OpenAI’s o3 obtained impossibly high scores by exploiting the scorer rather than solving the task — for instance monkey-patching the timing and synchronisation functions instead of writing a performant kernel — reward hacking in roughly 30% of runs, and continuing to do so even when explicitly instructed not to. Detecting these hacks is laborious and unreliable: METR resorts to manually inspecting anomalously high-scoring runs and to LLM-based monitors whose high false-positive rates require a further round of manual review, and they note that the discrepancies between the hacks each method surfaces suggest their counts “may be a significant underestimate”. ↩
This is generally my preferred method, because optimizing a prompt is like an infinite rabbit hole and often leads to quite ad-hoc and brittle improvements. ↩

Claude 4 hacked SWE-bench by peeking at future commits

2025-09-05T00:00:00+00:00

In a previous post about creating a Docker registry of SWE-bench images, I made a brief side note about hypothetical reward hacking:

[…] as a side note, it’s worth asking whether the git history should be included at all. It makes sense for the model to have access to the past git history, like a human developer would. However, the model should not have access to future history from after the PR was merged. A sophisticated cheating model could in theory reward hack the evaluation if there is any way to access future history. I believe that is possible in some circumstances even after a git reset --hard and git remote remove origin. For example, if version tags are used, the model could git checkout a future tag. Or it could access the dangling commits in some way, perhaps with git fsck --lost-found. I believe this is unlikely to be a problem currently, but could become one with future models.

Less than two months later, it seems that this behaviour has now been seen in the wild!

A META AI researcher has just posted an issue on the SWE-bench repo:

We’ve identified multiple loopholes with SWE Bench Verified where agents may look at future repository state (by querying it directly or through a variety of methods), and cases in which future repository state includes either solutions or detailed approaches to solving problems (commit messages and more).

… In a trajectory with Claude 4 Sonnet, pytest-dev__pytest-6202, the agent uses git log --all which leaks future commits that directly fix the issue.

Claude 4 Sonnet came out in May 2025, and I wrote my post in July. I was quite wrong about how long it would take for models to become sophisticated enough to cheat in this way. They were already doing it. I had never seen a model attempt this loophole, but one man can only read so many SWE-bench traces.

The command git log --all (which I didn’t know about) shows all commits reachable from any ref. A ref is any reference to the SHA-1 value of a commit, found under .git/refs, and includes: (1) the heads of branches, (2) tags, (3) remotes, and (4) any manually created refs. As I explained in the previous post, the original SWE-bench authors only removed the default remote, with a code comment: “Remove the remote so the agent won’t see newer commits”. But they didn’t think about other refs, like tags (a fix is underway).

git log --all lets a cheating model efficiently find any (non-dangling) commits, without having to look through the refs and figure out which one might point to future history and be usable for cheating. For example, Claude 4 Sonnet cleverly searched for commit messages with keywords relevant to the issue: git log --oneline --all | grep -i "bracket|parametrize|modpath".

In the pytest-6202 example, the issue is from November 2019. git log --all shows commits up to d0f136f (September 2024), which is tagged as 8.3.3. Indeed 8.3.3 seems to be the highest version tag present on the main branch:

$ git tag --list
1.0.0
1.0.0b3
1.0.0b6
...many more tags...
8.3.2
8.3.3
8.4.0.dev0
v7.4.3

In the example issues I looked at, git tags were the reason the future history was available. This is exactly the vulnerability that had jumped out at me when I looked into how the Docker images are constructed.

In pytest-6202, deleting all tags is sufficient to remove the offending commits from git log --all. However, a git fsck --lost-found will uncover the dangling tags and commits (ones not pointed to by any ref).

Here we must get into one subtlety. git show-ref 8.3.3 will show d430e32, not d0f136f, the actual September 2024 commit. d430e32 is the SHA-1 of the annotated tag object, which itself points to the actual commit d0f136f. git show-ref --dereference 8.3.3 can be used to show d0f136f.

Even after deleting tags, git fsck --lost-found will uncover

$ git fsck --lost-found
...
dangling tag d430e325c6d8c7161ae2e468ea5045a163e4c517
...

A cheater can then directly do git checkout d430e32, which moves the HEAD to the September 2024 commit d0f136f. Since HEAD is a ref, git log --all will at that point show the entire future history, just like before we deleted the tags (in fact you don’t need the --all now).

My understanding is that git gc --prune=now (docs) removes any unreachable objects. The documentation says:

git gc tries very hard not to delete objects that are referenced anywhere in your repository. In particular, it will keep not only objects referenced by your current set of branches and tags, but also objects referenced by the index, remote-tracking branches, reflogs (which may reference commits in branches that were later amended or rewound), and anything else in the refs/* namespace.

The reflog and index are strictly local, so there should be nothing in them just after cloning. We’ve covered branches, tags, and remotes. The only exception I can think of are manually created refs (using git update-ref), which should be extremely unusual. To mitigate this, one could trust only the HEAD ref and delete all others.

I spent a couple of hours today trying to find other loopholes that work even after git gc. I scoured various git internals I had never thought about before, like packfiles (in .git/objects/pack), or .git/objects/info/commit-graph. None of these worked. But git is a big program with a lot of arcane features, and I only tried a couple of SWE-bench issues. Can you find a more sophisticated hack?

How to run SWE-bench Verified in one hour on one machine

2025-07-10T00:00:00+00:00

I wrote this post for Epoch AI, where I work. You can also read it on the Epoch AI blog

We are releasing a public registry of Docker images for SWE-bench, to help the community run more efficient and reproducible SWE-bench evaluations. By making better use of layer caching, we reduced the total size of the registry to 67 GiB for all 2290 SWE-bench images (10x reduction), and to 30 GiB for 500 SWE-bench Verified images (6x reduction). This allows us to run SWE-bench Verified in 62 minutes on a single GitHub actions VM with 32 cores and 128GB of RAM.

Background

SWE-bench is a benchmark designed to evaluate large language models on real-world software engineering tasks. It consists of 2,294 GitHub issues from 12 popular Python repositories, paired with the actual pull requests that resolved those issues.

For each task, the AI system is given access to the repo in its state immediately before the pull request was merged, along with the issue description. The AI system must then modify the codebase to fix the problem. Success is measured by whether the AI system’s solution passes the test suite from after the pull request was merged—meaning the AI system must produce changes that satisfy the same tests that validated the human developer’s solution.

SWE-bench Verified is a human-validated subset of 500 problems from the original benchmark.

SWE-bench has traditionally been considered a challenging benchmark to run. For example, the All Hands team reports that it originally took them “several days” to run SWE-bench lite (300 examples), or over 10 minutes per sample. By using 32 machines in parallel, they reduced this to “several hours”.

SWE-bench and Docker

During SWE-bench evaluation, the AI system directly modifies a codebase in a sandbox with the necessary dependencies. Each question requires its own sandbox environment to capture the exact state of the repo immediately before the PR was merged, and these environments are specified using Docker images. The SWE-bench repo contains scripts that generate Dockerfiles and their context from the SWE-bench dataset, but without an image registry that stores the actual images, these images must be built from source.

There are two problems with building from source:

Speed: Building the images takes a long time. When operating on a development machine, you only need to build them once, after which they are cached. However, to run rigorously auditable evaluations at scale, like we do for our Benchmarking Hub, it’s necessary to use cloud-based VMs that are ephemeral. In this case, every evaluation run would require building all the images again.

Reproducibility: The build process relies on external resources (like apt or PyPI packages), so images built at different times may not be identical, even if the Dockerfiles and local context are the same. In particular, if the Dockerfiles do not pin versions for dependencies (which SWE-bench Dockerfiles generally do not, see below), the image will depend on the dependency resolution at build time. In addition, if a version of some dependency becomes unavailable, or moves to another address, etc., the build will fail. This may become an increasing problem over time, as many of the SWE-bench issues are now quite old (e.g. from 2015).

Therefore, our first contribution is a public registry of Docker images for SWE-bench, containing 2290 images for the x86_64 architecture (we were unable to build 4 out of 2294 images). This registry is free for anyone to use.

In addition, we optimized the images so that the total size of the registry is reduced by 6-10x. The rest of this post describes these optimizations.

Docker layering

The 2294 issues are from 12 repos, so many issues share the same repo. In fact, the distribution is heavily skewed, with Django accounting for 850/2294, or 37% of the issues.

Distribution of SWE-bench issues by repository, showing Django accounts for 850 out of 2294 total issues.

In addition, many issues will be close to each other in time, so the codebase will likely be similar. Two Django issues a week apart, for example, will have very similar dependencies. The main difference between them will be the change in application code between the two issues (usually a small fraction of the total codebase).

This is a perfect use case for Docker’s layer caching. What follows is a slightly simplified 1-paragraph explanation of layer caching. Each RUN, COPY, or ADD instruction in a Dockerfile creates a layer. A layer represents a specific modification to the container’s file system. Layers are added from bottom to top. Each layer is cached: if I change a layer, only layers above (that depend on it) will be rebuilt. In other words, if I make a change to a Dockerfile instruction and rebuild, this line and subsequent lines in the Dockerfile will have their layers re-built. Previous lines do not need to run again.¹

The SWE-bench Dockerfiles, created by Princeton researchers and OpenAI staff, do not make good use of layer caching and leave much low-hanging fruit for optimisation.

I will highlight just a few of the optimisations I made. To skip directly to overall results on size and runtime, click here. If you are interested in the full list of optimisations, it’s in the commit history of this repo.

Anatomy of a SWE-bench Dockerfile

Let’s take a closer look at a typical Django instance, django__django-13371.²

All SWE-bench images are built in three stages, base, env, and instance. Many instance images depend on a single env image. For example, the 850 Django images rely on 12 env images.

The Dockerfiles are all virtually identical and outsource the actual work to a setup_env.sh and setup_repo.sh script.

Here is the base Dockerfile:

# Base (ghcr.io/epoch-research/sweb-c7d4d9d4.base.x86_64)
FROM --platform=linux/x86_64 ubuntu:22.04

## ... long apt install command omitted ...

# Download and install conda
RUN wget 'https://repo.anaconda.com/miniconda/Miniconda3-py311_23.11.0-2-Linux-x86_64.sh' -O miniconda.sh \
    && bash miniconda.sh -b -p /opt/miniconda3
# Add conda to PATH
ENV PATH=/opt/miniconda3/bin:$PATH
# Add conda to shell startup scripts like .bashrc (DO NOT REMOVE THIS)
RUN conda init --all
RUN conda config --append channels conda-forge

RUN adduser --disabled-password --gecos 'dog' nonroot

And here is the env Dockerfile:

# env (ghcr.io/epoch-research/sweb-c7d4d9d4.env.x86_64.e83e37f52c09532c62acfb)
FROM --platform=linux/x86_64 ghcr.io/epoch-research/sweb-c7d4d9d4.base.x86_64:latest

COPY ./setup_env.sh /root/
RUN chmod +x /root/setup_env.sh
RUN /bin/bash -c "source ~/.bashrc && /root/setup_env.sh"

WORKDIR /testbed/

# Automatically activate the testbed environment
RUN echo "source /opt/miniconda3/etc/profile.d/conda.sh && conda activate testbed" > /root/.bashrc

Finally, here is the instance Dockerfile:

# instance (ghcr.io/epoch-research/sweb-c7d4d9d4.eval.x86_64.django__django-13371)
FROM --platform=linux/x86_64 ghcr.io/epoch-research/sweb-c7d4d9d4.env.x86_64.e83e37f52c09532c62acfb:latest

COPY ./setup_repo.sh /root/
RUN /bin/bash /root/setup_repo.sh

WORKDIR /testbed/

Side note: although we are here calling env and instance different ‘stages’, this is different from a genuine multi-stage build. For more detail, look at this footnote³.

Let’s look at the setup_env.sh and setup_repo.sh scripts for the Django 13371 instance.

This is the setup_env.sh script:

# setup_env.sh for `env` image e83e37f52c09532c62acfb
# (used by Django 13371)
set -euxo pipefail
source /opt/miniconda3/bin/activate
conda create -n testbed python=3.6 -y
cat <<'EOF_59812759871' > $HOME/requirements.txt
asgiref >= 3.3.2
argon2-cffi >= 16.1.0
...long list of dependencies truncated...
colorama; sys.platform == 'win32'
EOF_59812759871


conda activate testbed && python -m pip install -r $HOME/requirements.txt
rm $HOME/requirements.txt
conda activate testbed

And this is the setup_repo.sh script:

# setup_repo.sh for django__django-13371
set -euxo pipefail
git clone -o origin https://github.com/django/django /testbed
chmod -R 777 /testbed
cd /testbed
git reset --hard 3a9f192b131f7a9b0fe5783c684b23015fa67cc8
git remote remove origin
source /opt/miniconda3/bin/activate
conda activate testbed
echo "Current environment: $CONDA_DEFAULT_ENV"
python -m pip install -e .

There are numerous things to comment on here, but how do we find the most important optimisations to reduce the overall size of the images required to run SWE-bench?

We can use the fantastic tool dive (github.com/wagoodman/dive), which lets us see the layers of an image and their size. We can also see what actually changed on the filesystem in each layer with the file tree view on the right-hand side.

For example, here is the dive output for the Django 13371 image, focusing on the layer created by setup_env.sh:

The dive tool output for the Django 13371 image.

In the next section, I’ll explain how to interpret this output to suggest optimisations.

Moving the git clone operation

One salient feature of the output above can be seen from the layer sizes (top-left corner) alone: the topmost layer, corresponding to setup_repo.sh, is 330MB. Optimizations to this layer are tens of times more impactful than optimizations to the setup_env.sh layer, because the setup_repo.sh layer is different for each instance. (The ratio should be roughly 12:850 for the full SWE-bench, which is 70x, and still well over 10x for SWE-bench Verified)

setup_repo.sh does two major things in terms of disk space:

Cloning the Django repo with its full git history (git clone -o origin https://github.com/django/django /testbed)
Installing any additional dependencies required by this specific revision, that were not already present in the env image (python -m pip install -e .)

Which of these is more important? By looking at the layer contents, we can see that, in this case, very few other dependencies were needed, and cloning the repo took up virtually all the 330MB.

The dive output for the Django 13371 image, focused on the final (topmost) layer.

This represents an opportunity for optimization. We can move the git clone operation to the env image, so that it is shared across many Django instances. setup_repo.sh will only need to git reset --hard to the correct commit. This optimization is especially strong because the git history itself, stored in .git, represents 291MB of the 330MB layer size.

The new final layer is 40MB:

The optimized final layer reduced from 330MB to 40MB after moving the git clone operation to the env stage.

The diff between the two codebases, which represents the size of this layer, could no doubt be optimized much further. In this example the diff is the same size as the whole codebase (excluding the git history), because we are using a very naive approach: just cloning the latest version at the env stage. Instead, we could check out the appropriate revision that should be closer to the revision at the instance stage. However, for this project I kept the changes as simple as possible, in order to more easily be confident that the optimized instance images are still identical to the original ones.

Should the git history be included?

For this project I focused on pure size optimisations, because the images in the public registry should be identical to those that would be generated by the original build scripts from the SWE-bench authors. However, as a side note, it’s worth asking whether the git history should be included at all. It makes sense for the model to have access to the past git history, like a human developer would. However, the model should not have access to future history from after the PR was merged. A sophisticated cheating model could in theory reward hack the evaluation if there is any way to access future history. I believe that is possible in some circumstances even after a git reset --hard and git remote remove origin. For example, if version tags are used, the model could git checkout a future tag. Or it could access the dangling commits in some way, perhaps with git fsck --lost-found. I believe this is unlikely to be a problem currently, but could become one with future models.

The matplotlib 1.9 GB top layer

The next thing I noticed is that many matplotlib images were among the largest, often weighing in at 9GB. If we look at instance matplotlib__matplotlib-23913, we can see that setup_repo.sh is as follows:

# setup_repo.sh for matplotlib__matplotlib-23913
set -euxo pipefail
git clone -o origin https://github.com/matplotlib/matplotlib /testbed
chmod -R 777 /testbed
cd /testbed
git reset --hard 5c4595267ccd3daf78f5fd05693b7ecbcd575c1e
git remote remove origin
source /opt/miniconda3/bin/activate
conda activate testbed
echo "Current environment: $CONDA_DEFAULT_ENV"
apt-get -y update && apt-get -y upgrade && DEBIAN_FRONTEND=noninteractive apt-get install -y imagemagick ffmpeg texlive texlive-latex-extra texlive-fonts-recommended texlive-xetex texlive-luatex cm-super dvipng
python -m pip install -e .

The penultimate line installs a large number of heavy apt dependencies. This contributes to making the last layer 1,900 MB in size!

Should these commands be at the instance stage? They are taken from the pre_install key in a config dictionary that looks like this:

# swebench/harness/constants.py

# ... 
SPECS_MATPLOTLIB.update(
    {
        k: {
            "python": "3.8",
            "packages": "requirements.txt",
            "install": "python -m pip install -e .",
            "pre_install": [
                "apt-get -y update && apt-get -y upgrade && DEBIAN_FRONTEND=noninteractive apt-get install -y imagemagick ffmpeg libfreetype6-dev pkg-config texlive texlive-latex-extra texlive-fonts-recommended texlive-xetex texlive-luatex cm-super"
            ],
            "pip_packages": ["pytest", "ipython"],
            "test_cmd": TEST_PYTEST,
        }
        for k in ["3.1", "3.2", "3.3", "3.4"]
    }
)

SPECS_MATPLOTLIB.update(
    {
        k: {
            "python": "3.5",
            "install": "python setup.py build; python setup.py install",
            "pre_install": [
                "apt-get -y update && apt-get -y upgrade && apt-get install -y imagemagick ffmpeg"
            ],
            "pip_packages": ["pytest"],
            "execute_test_as_nonroot": True,
            "test_cmd": TEST_PYTEST,
        }
        for k in ["2.0", "2.1", "2.2", "1.0", "1.1", "1.2", "1.3", "1.4", "1.5"]
    }
)

The pre_install command only depends on the version k of matplotlib, not on the specific instance. Therefore, it makes sense to move this command to a deeper layer, i.e. to the env image. Doing so reduces the size of the topmost layer from 1,900 MB to 110MB, a 17x reduction.

Disabling the pip cache (or how to go insane)

A much simpler optimisation is to disable the pip cache. It’s standard practice in Dockerfiles to pass --no-cache-dir to pip, so that the pip cache is not stored in the image. I only bring this one up because it led me to an interesting rabbit hole.

There are pip commands in a number of places in the setup scripts. To keep diffs simple, I thought: better to use the PIP_NO_CACHE_DIR environment variable, rather than adding the --no-cache-dir flag to every pip command. This very reasonable idea was about to lose me a day of my life.

I set PIP_NO_CACHE_DIR=1 and tested it on a few images, and it worked as expected. However, when I wanted to rebuild all the images, I discovered that some would no longer build, with a mysterious traceback looking something like this:

Traceback (most recent call last):
  File "/local/lib/python3.5/site-packages/pip/_internal/basecommand.py", line 78, in _build_session
    if options.cache_dir else None
  File "/lib/python3.5/posixpath.py", line 89, in join
    genericpath._check_arg_types('join', a, *p)
  File "/lib/python3.5/genericpath.py", line 143, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'int'

I was baffled that the behaviour of pip could change between instances, because pip is one of the first things installed in the base image that all others inherit from:

RUN apt update && apt install -y \
...
python3 \
python3-pip 

The traceback did lead me to this ancient GitHub issue from 2018. The issue was fixed in this PR, which was merged into pip 19 and later.

However, my base image used a much more recent version of pip.

After much despair and gnashing of teeth, I eventually discovered that the images use Anaconda at the env stage like this:

# swebench/harness/test_spec.py
cmd = f"conda create -n {env_name} python={specs['python']} -y"

Anaconda installs different versions of pip depending on the Python version. The oldest issues in the SWE-bench dataset are very old. They use ancient versions of Python, and hence of pip. For example, Django 5470 is from 2015, and the corresponding image uses Python 3.5 and pip 10.

So, if PIP_NO_CACHE_DIR=1 crashes pip on pip < 19, how were we supposed to disable the cache? As I found out by GitHub comments from 2016, by setting PIP_NO_CACHE_DIR=0!

GitHub comment from 2016 expressing surprise that one needs to set PIP_NO_CACHE_DIR=0 to disable the pip cache.

The aforementioned PR (merged into pip 19) fixes the crash, and causes PIP_NO_CACHE_DIR=1 to disable the cache as expected. However, backward compatibility was preserved with the “well-known” behaviour that PIP_NO_CACHE_DIR=0 disables the cache.

GitHub comment from 2018 discussing backward compatibility.

In summary, I was able to disable the cache for all instances by setting PIP_NO_CACHE_DIR=0. Of course, all modern documentation on pip instructs you to set PIP_NO_CACHE_DIR=1 to disable the cache, and doesn’t mention this behaviour. So I had set up a devilishly confusing Dockerfile. I wrote a long code comment explaining the issue, and vowed to never do this again.

Impact on size

The 2294 SWE-bench docker images are sometimes reported to be around 2,000 GB in size. For example, the Princeton/OpenAI team wrote that

By default, the harness cache_level is set to env, which means that the harness will store the base and environment images, but not the instance images.

In this setting, the base and environment images will be reused across runs, but take up about 100GB of disk space. At the time of release, we require about 120GB of free disk space to run the harness with any cache_level. [Note by TA: they mean any cache_level other than instance]

For users who want the fastest possible evaluation times, we recommend setting cache_level to instance. In this setting, the harness will store images for all instances […] However, all base, environment, and instance images will be stored, taking up about 2,000GB of disk space.

The 120GB number (for a cache level other than instance) has been widely reported by evals professionals, for example:

Screenshot showing reported disk space requirements for SWE-bench evaluations.

Similarly, the All Hands team wrote that

because each agent runs in its own environment, it is necessary to create many environments, which requires hundreds of gigabytes of space

I’m not sure where the 2,000 GB number comes from. When summing the individual sizes of all (original) SWE-bench images, I get 3129 GiB. However, this ignores that many of these images share the same layers! When Docker builds or pulls these images, shared layers are only stored once.

When correctly calculated by summing only the sizes of the unique layers, the size of the unoptimized SWE-bench images comes to 684 GiB, which is nowhere near 2,000 GB. For SWE-bench Verified, the true size of the original images is 189 GiB. (You can reproduce my calculation using the script get_registry_size.py, and my data is also shared in the repository.)

After all my optimisations, the total size of the SWE-bench images is 67 GiB (10x reduction) while the SWE-bench Verified set fits in 30 GiB (6x reduction).

	SWE-bench (2290)	SWE-bench Verified (500)
Optimised (ours)	67 GiB	30 GiB
Original	684 GiB	189 GiB
Original, reported by SWE-bench authors	1,800 GiB (2,000 GB)	N/A

Running SWE-bench Verified in about an hour

Using our image registry, we’re able to run SWE-bench Verified in 62 to 73 minutes for many major models. Specifically, we ran the benchmark on a single GitHub actions runner with 32 cores and 128GB of RAM. We gave models a limit of 300,000 tokens per sample for the whole conversation (i.e. summing input and output tokens for every request in the conversation)⁴. Here were the runtimes for three major models:

gemini-2.0-flash-001: 62 minutes
gpt-4o-2024-11-20: 70 minutes
claude-3-7-sonnet-20250219: 63 minutes

As we have discussed above, OpenHands reported evaluation times of 10 minutes per sample on one machine, which they were able to reduce to about 20 seconds per sample by using 32 machines in parallel.

Using our optimized image registry, we achieve speeds of about 8 seconds per sample on a single large machine. This is 77x faster than OpenHands, and still 2.4 times faster than what OpenHands achieved with 32 machines. The comparison isn’t strictly a fair one; while OpenHands didn’t share details of their hardware, they were likely using less powerful machines.

Note that we have high API rate limits on these models, which are necessary to replicate these runtimes. Each eval used 100-150M tokens (of which the majority cached tokens), so we are using roughly 2M tokens per minute during the evaluation.

How to use our image registry

Our image registry is public, MIT-licensed, and hosted on GitHub Container Registry. Each image can be accessed by its name, and we follow the same naming pattern as the original SWE-bench authors. At the moment, all images only have the tag latest.

The naming format is ghcr.io/epoch-research/swe-bench.eval.., for example ghcr.io/epoch-research/swe-bench.eval.x86_64.astropy__astropy-13236.

For x86_64, we’re able to build 2290/2294 (99.8%) of the images, and all 500/500 out of the SWE-bench Verified set.

For arm64, 1819 (out of 2294) images are provided on a best-effort basis, and have not been tested.

Note the potentially confusing metaphor: an image is built by adding layers from bottom to top, so higher layers come later in the Dockerfile, while lower (deeper) layers come earlier in the Dockerfile. ↩
This example isn’t cherry-picked, I chose it at random towards the middle of the Django dataset, when sorted by issue number. (The issue numbers range from around 5000 to around 17,000, and the vast majority are in the 10,000 to 17,000 range.) ↩
As the docs explain, the purpose of a multi-stage build is to selectively copy artifacts from one stage to another:

With multi-stage builds, you use multiple FROM statements in your Dockerfile. Each FROM instruction can use a different base, and each of them begins a new stage of the build. You can selectively copy artifacts from one stage to another, leaving behind everything you don’t want in the final image.

In a multi-stage build, we selectively copy artifacts to make the final stage smaller and more cacheable. In SWE-bench, the stages just build on top of one another without any selective copying. From the point of view of caching and of the final images being produced, it is just as if we put all the instructions in one Dockerfile. The purpose of the three stages from the SWE-bench authors is likely to make the large number of generated Dockerfiles more manageable, and to be able to selectively prune just the instance stages while running the benchmark on a machine with limited disk space. ↩
The SWE-bench Verified evaluations on our Benchmarking Hub currently set this maximum to 1 million tokens. We also don’t run the eval directly on the images as they are in the registry, but add a few very lightweight layers containing the SWE-Agent tools that we use for our evaluations. For most models, the runtime in production is very similar to the numbers in this blog post; for some reasoning models that create a lot of output, with 1 million tokens we are bottlenecked by API rate limits. For more details on our evaluation setup, see the Benchmarking Hub. ↩

How should we analyse survey forecasts of AI timelines?

2024-12-16T00:00:00+00:00

Read at AI Impacts

The Expert Survey on Progress in AI (ESPAI) is a large survey of AI researchers about the future of AI, conducted in 2016, 2022, and 2023. One main focus of the survey is the timing of progress in AI¹.

The timing-related results of the survey are usually presented as a cumulative distribution function (CDF) showing probabilities as a function of years, in the aggregated opinion of respondents. Respondents gave triples of (year, probability) pairs for various AI milestones. Starting from these responses, two key steps of processing are required to obtain such a CDF:

Fitting a continuous probability distribution to each response
Aggregating these distributions

These two steps require a number of judgement calls. In addition, summarising and presenting the results involves many other implicit choices.

In this report, I investigate these choices and their impact on the results of the survey (for the 2023 iteration). I provide recommendations for how the survey results should be analysed and presented in the future.

Headline results

This plot represents a summary of my best guesses as to how the ESPAI data should be analysed and presented.

Thousands of AI Authors on the Future of AI, Figure 3. I added annotations to the 20%, 50%, and 80% points, for comparison with my plot.

I differ from previous authors in four main ways:

Show distribution of responses. Previous summary plots showed a random subset of responses, rather than quantifying the range of opinion among experts. I show a shaded area representing the central 50% of individual-level CDFs (25th to 75th percentile). More
Aggregate task and occupation questions. Previous analyses only showed task (HLMI) and occupation (FAOL) results separately, whereas I provide a single estimate combining both. By not providing a single headline result, previous approaches made summarization more difficult, and left room for selective interpretations. I find evidence that task automation (HLMI) numbers have been far more widely reported than occupation automation (FAOL). More
Median aggregation. I’m quite uncertain as to which method is most appropriate in this context for aggregating the individual distributions into a single distribution. The arithmetic mean of probabilities, used by previous authors, is a reasonable option. I choose the median merely because it has the convenient property that we get the same result whether we take the median in the vertical direction (probabilities) or the horizontal (years). More
Flexible distributions: I fit individual-level CDF data to “flexible” interpolation-based distributions that can match the input data exactly. The original authors use the Gamma distribution. This change (and distribution fitting in general) makes only a small difference to the aggregate results. More

Combined effect of our approach (3/4 elements), compared with previous results. For legibility, this does not show the range of responses, although I consider this one of the most important innovations over previous analyses.

CDF	Framing of automation	Distribution family	Loss function	Aggregation	p20	p50	p80
Blue (ours)	Aggregate of tasks (HLMI) and occupations (FAOL)	Flexible	Not applicable	Median	2048	2073	2103
Orange (previous)	Tasks (HLMI)	Gamma	MSE of probabilities	Arithmetic mean of probabilities	2031	2047	2110
Green (previous)	Occupations (FAOL)	Gamma	MSE of probabilities	Arithmetic mean of probabilities	2051	2110	2843

Note: Although previous authors give equal prominence to the orange (tasks, HLMI) and green (occupations, FAOL) results, I find evidence that the orange (tasks, HLMI) curve has been far more widely reported (More).

The last two points (aggregation and distribution fitting) directly affect the numerical results. The first two are about how the headline result of the survey should be conceived of and communicated.

These four choices vary in both their impact, and in my confidence that they represent an improvement over previous analyses. The two tables below summarise my views on the topic.

Choice	Impact on understanding and communication of main results	Confidence it’s an improvement
Show range of responses (More)	High	Very high
Aggregate FAOL and HLMI (More)	High	Moderate

Choice	Numerical impact on aggregate CDF	Confidence it’s an improvement
Median aggregation (More)	High	Very low
Flexible distributions (More)	Minimal	High

Even if you disagree with these choices, you can still benefit from my work! The code used to implement these new variations is open source. It provides user-friendly configuration objects that make it easy to run your own analysis and produce your own plots. The source data is included in version control. AI Impacts plans to use this code when analysing future iterations of ESPAI. I also welcome engagement from the wider research community.

Suggested textual description

If you need a textual description of the results in the plot, I would recommend:

Experts were asked when it will be feasible to automate all tasks or occupations. The median expert thinks this is 20% likely by 2048, and 80% likely by 2103. There was substantial disagreement among experts. For automation by 2048, the middle half of experts assigned it a probability between 1% and a 60% (meaning ¼ assigned it a chance lower than 1%, and ¼ gave a chance higher than 60%). For automation by 2103, the central half of experts forecasts ranged from a 25% chance to a 100% chance.²

This description still contains big simplifications (e.g. using “the median expert thinks” even though no expert directly answered questions about 2048 or 2103). However, it communicates both:

The uncertainty represented by the aggregated CDF (using the 60% belief interval from 20% to 80%)
The range of disagreement among experts (using the central 50% of responses)

In some cases, this may be too much information. I recommend if at all possible that the results should not be reduced to the single number of the year by which experts expect a 50% chance of advanced AI. Instead, emphasise that we have a probability distribution over years by giving two points on the distribution. So if a very concise summary is required, you could use:

Surveyed experts think it’s unlikely (20%) it will become feasible to automate all tasks or occupations by 2048, but it probably will (80%) by 2103.

If even greater simplicity is required, I would urge something like the following, over just using the median year:

AI experts think full automation is most likely to become feasible between 2048 and 2103.

Headline results
1. Suggested textual description
The distribution of raw responses
1. Example: Retail Salesperson occupation
2. Timing of human-level performance
Aggregation
Distribution fitting
Displaying the distribution of responses
Aggregating across the task and occupation framings
Codebase

The distribution of raw responses

Even readers who are familiar with ESPAI may only have seen the results after processing. It can be helpful to look at the raw data, i.e. respondent’s answers to questions before any processing, to remind ourselves how the survey was conducted.

All questions about how soon a milestone would be reached were framed in two ways: fixed-years and fixed-probabilities. Half of respondents were asked to estimate the probability that a milestone would be reached by a given year (“fixed-years framing”), while the other half were asked to estimate the year by which the milestone would be feasible with a given probability (“fixed-probabilities framing”).

Example: Retail Salesperson occupation

Responses about one such milestone (say, the occupation of retail salesperson in the example below), if shown as a scatterplot, form three horizontal lines for fixed probabilities, and three vertical lines for fixed years. These correspond to the six questions being asked about the milestone:

The scatterplot is a helpful reminder of the data’s shape in its rawest form. However, all scatterplots will form three horizontal and three vertical lines. We can show more structured information about the distribution of responses for each of the six questions by using six box and whisker plots³, as shown below:

We can see several useful things in this set of box plots:

There is a large framing effect, whereby the fixed-years framing produces later predictions. (This effect is familiar from previous analyses of ESPAI, where it has been shown to occur systematically).
- For example, the prediction (2043, 50%) is the 85th percentile of responses for the 50% question in the fixed-probabilities framing, while the same prediction is the median response for the 2043 question in the fixed-years framing.
- When asked about 2073 in the fixed-years framing, the median response was 90%, which is much later than even the 85th percentile response to the 90% question int the fixed-probabilities framing.
Responses follow a skewed distribution
- For all three questions in the fixed-probabilities framing, the responses have a large right skew
- In the fixed-years framing, the 2033 question produces a right skew (up skew in the boxplot), whereas the 2073 question produces a left-skew (down skew in the boxplot), with more than 25% of respondents giving a probability of 100%.
There is a wide range of responses, indicating substantial disagreement among respondents. For example, when asked about 2043, the interval (centred on the median) that contains half of responses ranged from a 30% chance to a 90% chance. The interval that contains 70% of responses ranged from a 10% chance to a 98% chance.

We can now look at the distribution of raw responses for the timing of human-level performance.

Timing of human-level performance

When the survey investigated the timing of human-level performance, the question was framed in two ways, as tasks, and as occupations⁴:

“High-Level Machine Intelligence” (HLMI): when unaided machines can accomplish every task better and more cheaply than human workers.
“Full Automation of Labor” (FAOL): when for any occupation, machines could be built to carry it out better and more cheaply than human workers.

We can now take each of these in turn (expand the collapsible sections below).

Raw data

In the fixed probabilities framing, respondents were asked for the number of years until a 10%, 50%, and 90% probability of FAOL.

Probability of FAOL	Mean response	15th percentile response	Median response	85th percentile response
10%	508177	10	40	100
50%	783590	20	70	200
90%	1.01197e+06	35	100	500

In the fixed years framing, respondents were asked for the probability of FAOL within 10, 20, and 50 years.

Years until FAOL	Mean response	15th percentile response	Median response	85th percentile response
10	6.02%	0.00%	0.00%	10.00%
20	12.30%	0.00%	2.00%	30.00%
50	24.66%	0.00%	10.00%	60.00%

In the fixed probabilities framing, respondents were asked for the number of years until a 10%, 50%, and 90% probability of HLMI.

Probability of HLMI	Mean response	15th percentile response	Median response	85th percentile response
10%	41.19	2	5	20
50%	1307.74	7	20	50
90%	457537	15	50	100

In the fixed years framing, respondents were asked for the probability of HLMI within 10, 20, and 40 years.

Years until HLMI	Mean response	15th percentile response	Median response	85th percentile response
10	18.25%	0.00%	10.00%	50.00%
20	34.70%	4.00%	30.00%	75.00%
40	54.58%	10.00%	50.00%	95.00%

Aggregation

Possible methods

All previous analyses produced the aggregate distribution by taking the average of CDF values, that is, by taking the mean of probability values at each year.

There are many other possible aggregation methods. We can put these into two categories:

vertical methods like the above aggregate probability values at each year
horizontal methods aggregate year values at each probability

This figure illustrates both methods on a very simple example with two CDFs to aggregate.

For both vertical and horizontal aggregation, we need not take the mean of values. In principle any aggregation function could be used, of which the mean and median are only the two most obvious examples.

When it comes to aggregating probabilities (vertical aggregation), there are additional complications. The topic has been well studied, and many aggregation methods have been proposed.

A full assessment of the topic would take us far beyond the scope of this report, so I will only briefly mention one prominent recommendation: taking the geometric mean of odds. The core observation is that the arithmetic mean of probabilities ignores information from extreme predictions. This can be seen with a simple example. In scenario A, we aggregate the two predictions (1%, 10%), whereas in scenario B the two predictions are (0.1%, 10%). The arithmetic mean of probabilities is close to 5% in both cases (5.5% for A and 5.05% for B). It gives very little weight to the difference between 1% and 0.1%, which is after all a factor of 10. The geometric mean of odds reacts much more strongly to this much more extreme prediction, being about 3.2% in scenario A, and 1.0% in scenario B.

This behaviour of the geometric mean of odds is theoretically appealing, but it is only advisable if extreme predictions are really to be taken at face value. We might worry that these predictions are much more overconfident.

As a further complication, in the case of ESPAI, in practise we cannot apply the geometric mean of odds. This is because for nearly every year (every vertical line) we might consider, many of the respondents’ fitted CDFs take values indistinguishable from 0 or 1. This causes the geometric mean of odds to immediately become 0 or 1⁵.

Aggregating years is also problematic. Because the input is bounded on the left but not the right, the arithmetic mean of responses is inevitably dominated by extremely large values. This method would produce a CDF where any probability of the event is essentially infinitely many years away. We might hope to address this problem by using the geometric mean of years, but this in turn suffers from numerical and conceptual issues similar to those of the geometric mean of odds. Ultimately, taking the median of years is the only method of aggregating years I was able to apply.

The median of years and median of probabilities give the same answer. This makes intuitive sense since CDFs are strictly increasing⁶. So I simply call this the “median”.

Mean vs Median aggregation

As a result of these difficulties, I will present only the following aggregation methods:

(Arithmetic) mean of probabilities
Median of probabilities

These plots use the Gamma distribution with the mean square error (MSE) of probabilities as the loss function, so the mean aggregation line corresponds to the results of previous analyses.

Mean vs Median aggregation

We see a notable pattern in each of these cases. As we go from left to right, the median always starts below the mean (i.e. the median initially gives later predictions), but eventually overtakes the mean (i.e. the median eventually gives earlier predictions). Median aggregation also always gives rise to a more confident probability distribution: one whose probability mass is more concentrated.

Why the mean and median differ

When the mean is very different from the median, this means that the distribution of responses is highly skewed. We can illustrate this by displaying a histogram of CDF values for a given year.

The results for automation of all occupations (FAOL) are quite interesting. For the years 2040 and 2060, the results are extremely skewed. A large majority assigns very low probabilities, but there is a right tail of high probabilities, which causes the mean to greatly exceed the median. For the years 2080 and 2100, a bimodal distribution emerges. We have a big cluster with probabilities near zero and a big cluster with probabilities near 1. Opinion is extremely polarised. By 2200 the median exceeds the mean. When we reach 2500, a majority think FAOL is near-certain, but a significant left tail causes the mean to lag far behind the median.

With task automation (HLMI), we see the same basic pattern (repeated everywhere): the median at first trails behind the mean, and for later years and higher probabilities, the median overtakes the mean. However, skewness is less extreme than for FAOL, and we do not see a strongly bimodal histogram (extreme polarisation) at any point.

Aside: the winsorized geometric mean of odds

There is one way to use the geometric mean of odds that avoids the problem of zeroes and ones. This is to winsorize the data: to replace the most extreme values with less extreme values. For example, we could replace all values less than 0.1 with 0.1, and all values greater than 0.9 with 0.9.

Of course, this introduces a highly subjective choice that massively affects the results. We could replace all values less than 0.01 with 0.01, or all values less than 0.001 with 0.001. Therefore, I do not consider this technique suitable for the creation of headline result.

However, it lets us do some potentially interesting explorations. Winsorising essentially means we do not trust the most extreme predictions. We can now explore what the geometric mean of odds would look like under various degrees of winsorisation. e.g. what does it look like if we ignore all predictions more extreme than 1:100? What about 1:1000?

Very informally, we can see that for HLMI (tasks) and FAOL (occupations), the arithmetic mean of probabilities roughly corresponds to the geometric mean of odds with a winsorisation level of about 1:10. We’ve already discussed the well-known effect that the arithmetic mean of probabilities ignores more extreme predictions, compared to the geometric mean of odds. For this particular data, we can roughly quantify this effect, and see that it is equivalent to ignoring all predictions <10% and >90%. I find this to be quite an extreme level of winsorisation. Consider, for example, that the fixed-probabilities framing explicitly asked for predictions at the 10% and 90% levels – it would be odd to simultaneously consider these probabilities too extreme to be trusted.

Distribution fitting

All previous analyses of the ESPAI data fitted each respondent’s CDF data (triples of (year, probability)) to a Gamma distribution before aggregating these distributions.

Why fit a distribution?

Creating a full continuous distribution from three CDF points necessarily imposes some assumptions that were not present in the data. And recall, the respondents just gave numbers in text fields, and never saw the distribution that was later fitted to their CDF data.

So to begin with, it’s worth asking: why fit a distribution at all?

If we are only looking at a particular framing and question, for example FAOL with fixed years, it may indeed be preferable to look directly at the raw data. This allows us to talk strictly about what respondents said, without any additional assumptions. Even in this restricted setting, however, we might want to be able to get predictions for other years or probabilities than those respondents were asked about; this requires a full CDF. A simple example where this is required is for making comparisons across different iterations of the ESPAI survey in the fixed-years setting. Each survey asks for predictions about a fixed number of years from the date of the survey. So the fixed-years question asks about different calendar years each time the survey is made.

A more fundamental problem for the raw data approach is that we wish to aggregate the results of different framings into a single estimate. We can only aggregate across the fixed-years and fixed-probability framings by aggregating full distributions. In addition, even within the fixed-years framing, we cannot aggregate the occupations (FAOL) and tasks (HLMI) framings, because different years were used (10, 20, and 40 years for HLMI and 10, 20, and 50 years for FAOL).

Limitations of previous analyses

Constraints of Gamma distribution

While creating a full CDF from three points inevitably imposes assumptions not present in the data, we might think that, at a minimum, it would be desirable to have this CDF pass through the three points.

Previous analyses used a Gamma distribution. The gamma distribution is a 2-parameter distribution that is able to exactly match 2 points of CDF data, but not 3 points. The gamma distribution (and any 2-parameter distribution) necessarily loses information and distorts the expert’s beliefs even for the points where we know exactly what they believe.

Here are 9 examples from the fixed-probabilities framing⁷. They are representative of the bottom half of fits (from the 50th to 90th percentile). Each subplot shows an example where the fitted gamma CDF (shown as a gray curve) attempts to match three points from a respondent’s data (shown as red crosses).

First, we can see that the gamma is not flexible enough to match the three points. While the median fit (A1) is acceptable, some fits are poor.

Inappropriate loss function

In addition, if we look carefully we can see an interesting systematic pattern in the poor fits. We can see that when the Gamma has trouble fitting the data, it prefers to fit two points well, even at the expense of a very poor fit on the third point, rather than choosing a middle ground with an acceptable fit on all three points. This begins to be visible in row B (67th to 87th percentile), and becomes blatantly clear in row C (84th to 95th percentile). In fact, in C2 and C3, the Gamma fits two points exactly and completely ignores the third. When this happens, the worst-fit point is always the 0.9 or 0.1 point, never the 0.5 point.

The errors at the 0.1 and 0.9 points can completely change the nature of the prediction. This becomes clear if we go to odds space, and express the odds ratio between the data and the gamma CDF.

While a 2x odds ratio (e.g. in B2) is already substantial, when we move to the worst 15% of the fits, the odds ratio for the worst of the three points becomes astronomical.

The reason this happens is that the loss function used in previous work is not the appropriate one.

Previous analyses used mean squared error (MSE) of probabilities as their loss function: $L_{\text{MSE}} = \sum_i (p_i - \hat{p}_i)^2$ , where $p_i$ are the probabilities from the respondent’s data and $\hat{p}_i$ are the probabilities from the fitted CDF. This loss function treats all probability differences equally, regardless of where they occur in the distribution. For instance, it considers a deviation of 0.05 to be equally bad whether it occurs at $p=0.5$ or at $p=0.99$ .

This is inappropriate when fitting CDF data. Consider the case depicted in C2, where the respondent thinks the event in question is 90% likely by 150 years from the date of the survey. Meanwhile, the Gamma CDF fitted by MSE gives a probability of 99.98% at 150 years. This dramatic departure from the respondent’s beliefs is represented in the 777x odds ratio. A 777x odds ratio at $p=0.5$ would mean changing from even odds (1:1) to odds of 777:1, or a probability of >99.8%. (A 13x odds ratio, as seen for the 0.1 point in C1 (84th percentile), would mean changing from even odds to odds of 13:1, or a probability of 93%.)

The appropriate loss function for CDF data is the log loss, also known as the cross-entropy loss: $L_{\text{log}} = -\sum_i [p_i \log(\hat{p}_i) + (1-p_i)\log(1-\hat{p}_i)]$ . This loss function naturally accounts for the fact that probability differences near 0 and 1 represent much larger differences in beliefs than the same probability differences near 0.5.⁸

As expected from this theoretical argument, we can see that the log loss, unlike the MSE of probabilities, does not display the pathological behaviour of ignoring the 0.1 or 0.9 point, and so avoids extreme odds ratios (see especially C1-C3):

As an informal analysis, this plot suggests that the MSE leads to extremely poor fits on >15% of the data, but also that most of the MSE fits are close to the log loss fits.

When we create the aggregate CDF, we see hardly any impact of the loss function:

Flexible distributions

Regardless of the loss function used, we know that the Gamma distribution cannot match the three points of CDF data given by the expert⁹. When we use any such distribution, our results do not merely reflect the expert’s beliefs, they also reflect the mathematical constraint we imposed upon those beliefs.

Overriding expert responses in this way may be appropriate when we have a strong theoretical justification to impose a particular distribution family. For example, if we have a strong reason to believe that experts think (or ought to think) of a variable as a sum of many small independent contributions, we may wish to impose a normal distribution, even if the responses they gave us are incompatible with a normal distribution.

However, the authors of previous analyses did not justify the choice of the gamma distribution at any point. In addition, I am not aware of any strong argument to impose a particular distribution family in this case.

While creating a full CDF from three points inevitably imposes assumptions not present in the data, at a minimum, it would be desirable to have this CDF pass through the three points.

To achieve this, I used proprietary probability distributions which I call ‘flexible distributions’. I developed these over the last several years, for precisely the class of use cases faced by ESPAI. These distributions have the following properties:

Always exactly match three CDF points (or indeed an arbitrary number of them)…
…while taking a simple and smooth shape
Can be unbounded, or given an upper or lower bound, or both

The distributions I used in this analysis are based on interpolation theory. The full mathematical and algorithmic details are proprietary, and the distributions are available as a service at MakeDistribution.com. However, you can see how these distributions behave with the free interactive web UI below (select interpolation-based families under expert settings). In addition, to make this work reproducible, the specific fitted CDFs used in the ESPAI analysis are open source¹⁰.

Try the MakeDistribution interface

Elevated computing

2023-08-03T00:00:00+00:00

I made a mount to lift my laptop off the surface of my desk:

I operate my laptop with its lid closed, using only an external monitor. The downside is that the closed laptop is a useless rectangle that takes up a lot of precious desk space. This type of product would have solved 80% of the problem:

However, even more space could be saved by removing the laptop from the desk entirely. A pole or stand of some sort is already needed anyway to hold up the monitor, so why not use it to hold the laptop as well?

My solution consists of two repurposed commercially available products, along with a piece of 6mm birch plywood to hold them together.

Attached to the monitor pole, we have a VESA mount that can tilt and swivel. Instead of holding a monitor, it’s holding the wooden plank.

On the other side of the plank, we have 4 pieces of metal sold as an under-desk laptop mount:

Details

To keep the setup compact, we want the pole-to-VESA piece to be as short as possible. There are zillions of pole-mounted VESA monitor arms to choose from, but finding something without the arm was a bit more difficult.

All inputs were sourced from the internet:

I ordered the wood cut to measure from woodsheets.com , which cost £8.60 including shipping.
The pole mount was £13.77 from Amazon
The under-desk mounting solution was £16.99 from Amazon

I used linseed oil to give the wood a nicer finish; this is of course optional but the aesthetic improvement is significant.

The “under-desk” bits come with little silicon-ish pads to avoid scratching the laptop. Similarly, I put plastic caps on the screw heads on the laptop side:

Had I realised these would be needed, I would probably have chosen to use glue instead of screws to attach the wood to the VESA mount.

After completing this project, I also mounted a 3.5 inch external hard drive enclosure in a similar way. I glued it directly to the VESA mount. Arbitrary objects could be thus held aloft. The scope of this innovation has no limits currently known to science.

As I have two poles, I put the hard drive on the other pole, but it would have comfortably fit on the first pole.

Previous versions of this idea

For my previous attempt, I hand-sawed a piece of scrap wood from a previous project. This looked tolerable, but for £8.60 it was worth having another piece cut to size:

My initial idea was to use a VESA mount and a cheap laptop sleeve. I used this for several months:

However, inserting and removing the laptop was rather inconvenient because the sleeve lacks rigidity. With the wood-based solution, it’s a simple one-handed process. The sleeve also has worse heat dissipation, despite the addition of aeration holes.

How to run Cronicle (a cron replacement) in a Docker container

2022-05-07T00:00:00+00:00

I really don’t like cron jobs and crontab:

crontab has a horrible syntax (it’s from 1975…)
logging the output of jobs needs to be specified manually
viewing logs is inconvenient (even just for checking whether a job ran or not!)
cron jobs run in a minimal environment that’s inconvenient to modify

Cronicle is a friendlier alternative (“a task scheduler with a web based front-end UI”).

Very important: the default username/password for the web interface is admin/admin. This could let anyone run arbitrary shell commands on your server! Change the password immediately after setting up.

Here’s how to run Cronicle Dockerized.

docker run -d \
  -v /cronicle-data/data:/opt/cronicle/data:rw \
  -v /cronicle-data/logs:/opt/cronicle/logs:rw \
  -v /cronicle-data/plugins:/opt/cronicle/plugins:rw \
  -v /cronicle-data/app:/app:rw \
  --hostname your_hostname.com -p 11531:3012 \
  -e CRONICLE_base_app_url='http://your_hostname.com:11531' \
  --name cronicle \
  bluet/cronicle-docker:latest

You can now point http://your_hostname.com to the correct IP address and visit http://your_hostname.com:11531 in your browser to access a web interface for Cronicle.

Comments:

11531 is a port number chosen randomly. You should generally use 80, the default port used by web browsers; for me that port is occupied by other applications I run on the host web server.
The source for the Docker image bluet/cronicle-docker is here.
On 7 May 2022, I confirmed that these steps work on brand new server, with revision 3e4211e of bluet/cronicle-docker.

How much of the fall in fertility could be explained by lower mortality?

2021-08-05T00:00:00+00:00

Many people think that lower child mortality causes fertility to decline.

One prominent theory for this relationship, as described by Our World in Data¹, is that “infant survival reduces the parents’ demand for children”². (Infants are children under 1 years old).

In this article, I want to look at how we can precisify that theory, and what magnitude the effect could possibly take. What fraction of the decline in birth rates could the theory explain?

Important. I don’t want to make claims here about how parents actually make fertility choices. I only want to examine the implications of various models, and specifically how much of the observed changes in fertility the models could explain.

Constant number of children

One natural interpretation of “increasing infant survival reduces the parents’ demand for children” is that parents are adjusting the number of births to keep the number of surviving children constant.

Looking at Our World in Data’s graph, we can see that in most of the countries depicted, the infant survival rate went from about 80% to essentially 100%. This is a factor of 1.25. Meanwhile, there were 1/3 as many births. If parents were adjusting the number of births to keep the number of surviving children constant, the decline in infant mortality would explain a change in births by a factor of 1/1.25=0.8, a -0.2 change that is only 30% of the -2/3 change in births.

The basic mathematical reason this happens is that even when mortality is tragically high, the survival rate is still thankfully much closer to 1 than to 0, so even a very large proportional fall in mortality will only amount to a small proportional increase in survival.

Some children survive infancy but die later in childhood. Although Our World in Data’s quote focuses on infant mortality, it makes sense to consider older children too. I’ll look at under-5 mortality, which generally has better data than older age groups, and also captures a large fraction of all child mortality³.

England (1861-1951)

England is a country with an early demographic transition and good data available.

Doepke 2005 quotes the following numbers:

	1861	1951
Infant mortality	16%	3%
1-5yo mortality	13%	0.5%
0-5 yo mortality	27%	3.5%
Survival to 5 years	73%	96.5%
Fertility	4.9	2.1

Fertility fell by 57%, while survival to 5 years rose by 32%. Hence, if parents aim to keep the number of surviving children constant, the change in child survival can explain 43%⁴ of the actual fall in fertility. (It would have explained only 23% had we erroneously considered only the change in infant survival.)

Sub-Saharan Africa (1990-2017)

If we look now at sub-Saharan Africa data from the World Bank, the 1990-2017 change in fertility is from 6.3 to 4.8, a 25% decrease, whereas the 5-year survival rate went from 0.82 to 0.92, a 12% increase. So the fraction of the actual change in fertility that could be explained by the survival rate is 44%. (This would have been 23% had we looked only at infant survival).

Source data and calculations

So far, we have seen that this very simple theory of parental decision-making can explain 30-44% of the decline in fertility, while also noticing that considering childhood mortality beyond infancy was important to giving the theory its full due.

However, in more sophisticated models of fertility choices, the theory looks worse.

A more sophisticated model of fertility decisions

Let us imagine that instead of holding it constant, parents treat the number of surviving children as one good among many in an optimization problem.

An increase in the child survival rate can be seen as a decrease in the cost of surviving children. Parents will then substitute away from other goods and increase their target number of surviving children. If your child is less likely to die as an infant, you may decide to aim to have more children: the risk of experiencing the loss of a child is lower.⁵

For a more formal analysis, we can turn to the Barro and Becker (1989) model of fertility. I’ll be giving a simplified version of the presentation in Doepke 2005.

In this model, parents care about their own consumption as well as their number of surviving children. The parents maximise⁶

U(c,n) = u(c) + n^\epsilon V

where

$n$ is the number of surviving children and $V$ is the value of a surviving child
$\epsilon$ is a constant $\in (0,1)$
$u(c)$ is the part of utility that depends on consumption⁷

The income of a parent is $w$ , and there is a cost per birth of $p$ and an additional cost of $q$ per surviving child⁸. The parents choose $b$ , the number of births. $s$ is the probability of survival of a child, so that $n=sb$ .

Consumption is therefore $c=w-(p+qs)b$ and the problem becomes $\max_{b} U = u(w-(p+qs)b) + (sb)^\epsilon V$

Letting $b^{*}(s)$ denote the optimal number of births as a function of $s$ , what are its properties?

The simplest one is that $sb^*(s)$ , the number of surviving children, is increasing in $s$ . This is the substitution effect we described intuitively earlier in this section. This means that if $s$ is multiplied by a factor $x$ (say 1.25), $b^*(s)$ will be multiplied more than $1/x$ (more than 0.8).

When we looked at the simplest model, with a constant number of children, we guessed that it could explain 30-44% of the fall in fertility. That number is a strict upper bound on what the current model could explain.

What we really want to know, to answer the original question, is how $b^*(s)$ itself depends on $s$ . To do this, we need to get a little bit more into the relative magnitude of the cost per birth $p$ and the additional cost $q$ per surviving child. As Doepke writes,

If a major fraction of the total cost of children accrues for every birth, fertility [i.e. $b^*(s)$ ] would tend to increase with the survival probability; the opposite holds if children are expensive only after surviving infancy⁹.

This tells us that falling mortality could actually cause fertility to increase rather than decrease.¹⁰

To go further, we need to plug in actual values for the model parameters. Doepke does this, using numbers that reflect the child mortality situation of England in 1861 and 1951, but also what seem to be some pretty arbitrary assumptions about the parent’s preferences (the shape of $u$ and the value of $\epsilon$ ).

With these assumptions, he finds that “the total fertility rate falls from 5.0 (the calibrated target) to 4.2 when mortality rates are lowered to the 1951 level”¹¹, a 16% decrease. This represents is 28% of the actually observed fall in fertility to 2.1.

Extensions of Barro-Becker model

The paper then considers various extensions of the basic Barro-Becker model to see if they could explain the large decrease in fertility that we observe.

For example, it has been hypothesized that when there is uncertainty about whether a child will survive (hitherto absent from the models), parents want to avoid the possibility of ending up with zero surviving children. They therefore have many children as a precautionary measure. Declining mortality (which reduces uncertainty since survival rates are thankfully greater than 0.5) would have a strong negative impacts on births.

However, Doepke also considers a third model, that incorporates not only stochastic mortality but also sequential fertility choice, where parents may condition their fertility decisions on the observed survival of children that were born previously. The sequential aspect reduces the uncertainty that parents face over the number of surviving children they will end up with.

The stochastic and sequential models make no clear-cut predictions based on theory alone. Using the England numbers, however, Doepke finds a robust conclusion. In the stochastic+sequential model, for almost all reasonable parameter values, the expected number of surviving children still increases with $s$ (my emphasis):

To illustrate this point, let us consider the extreme case [where] utility from consumption is close to linear, while risk aversion with regards to the number of surviving children is high. … [W]hen we move (with the same parameters) to the more realistic sequential model, where parents can replace children who die early, … despite the high risk aversion with regards to the number of children, total fertility drops only to 4.0, and net fertility rises to 3.9, just as with the benchmark parameters. … Thus, in the sequential setup the conclusion that mortality decline raises net fertility is robust to different preference specifications, even if we deliberately emphasize the precautionary motive for hoarding children.

So even here, the fall in mortality would only explain 35% of the actually observed change in fertility. It seems that the ability to “replace” children who did not survive in the sequential model is enough to make its predictions pretty similar to the simple Barro-Becker model.

The quote in context on Our World in Data’s child mortality page: “the causal link between infant [<1 year old] survival and fertility is established in both directions: Firstly, increasing infant survival reduces the parents’ demand for children. And secondly, a decreasing fertility allows the parents to devote more attention and resources to their children.” ↩
As an aside, my impression is that if you asked an average educated person “Why do women in developing countries have more children?”, their first idea would be: “because child mortality is higher”. It’s almost a trope, and I feel that it’s often mentioned pretty glibly, without actually thinking about the decisions and trade-offs faced by the people concerned. That’s just an aside though – the theory clearly has prima facie plausibility, and is also cited in serious places like academia and Our World in Data. It deserves closer examination. ↩
It should be possible to conduct the Africa analysis for different ages using IMHE’s more granular data, but it’s a bit more work. (There appears to be no direct data on deaths per birth as opposed to per capita, and data on fertility is contained in a different dataset from the main Global Burden of Disease data.) ↩
All things decay. Should this Google Sheets spreadsheet become inaccessible, you can download this .xlsx copy which is stored together with this blog. ↩
In this light, we can see that the constant model is not really compatible with parents viewing additional surviving children as a (normal) good. Nor of course is it compatible with viewing children as a bad, for then parents would choose to have 0 children. Instead, it could for example be used to represent parents aiming for a socially normative number of surviving children. ↩
I collapse Doepke’s $\beta$ and $V$ into a single constant $V$ , since they can be treated as such in Model A, the only model that I will present mathematically in this post. ↩
Its actual expression, that I omit from the main presentation for simplicity, is $u(c)=\frac{c^{1-\sigma}}{1-\sigma}$ , the constant relative risk-aversion utility function. ↩
There is nothing in the model that compels us to call $p$ the “cost per birth”, this is merely for ease of exposition. The model itself only assumes that there are two periods for each child: in the first period, costing $p$ to start, children face a mortality risk; and in the second period, those who survived the first face zero mortality risk and cost $q$ . ↩
Once again, Doepke calls the model’s early period “infancy”, but this is not inherent in the model. ↩
It’s difficult to speculate about the relative magnitude of $p$ and $q$ , especially if, departing from Doepke, we make the early period of the model, say, the first 5 years of life. If the first period is only infancy, it seems plausible to me that $q \gg p$ , but then we also fail to capture any deaths after infancy. On the other hand, extending the early period to 5 incorrectly assumes that parents get no utility from children before they reach the age of 5. ↩
The following additional context may be helpful to understand this quote:

The survival parameters are chosen to correspond to the situation in England in 1861 . According to Perston et al. (1972) the infant mortality rate (death rate until first birthday) was $16 \%$ , while the child mortality rate (death rate between first and fifth birthday) was $13 \%$ . Accordingly, I set $s_{i}=0.84$ and $s_{y}=0.87$ in the sequential model, and $s=s_{i} s_{y}=0.73$ in the other models. Finally, the altruism factor $\beta$ is set in each model to match the total fertility rate, which was $4.9$ in 1861 (Chenais 1992). Since fertility choice is discrete in Models B and C, I chose a total fertility rate of $5.0$ as the target.

Each model is thus calibrated to reproduce the relationship of fertility and infant and child mortality in 1861 . I now examine how fertility adjusts when mortality rates fall to the level observed in 1951 , which is $3 \%$ for infant mortality and $0.5 \%$ for child mortality. The results for fertility can be compared to the observed total fertility rate of $2.1$ in 1951 .

In Model A (Barro-Becker with continuous fertility choice), the total fertility rate falls from $5.0$ (the calibrated target) to $4.2$ when mortality rates are lowered to the 1951 level. The expected number of surviving children increases from $3.7$ to $4.0$ . Thus, there is a small decline in total fertility, but (as was to be expected given Proposition 1) an increase in the net fertility rate.

↩

The special case of the normal likelihood function

2021-07-31T00:00:00+00:00

Summary¹: The likelihood function implied by an estimate $b$ with standard deviation $\sigma$ is the probability density function (PDF) of a $\mathcal{N}(b,\sigma^2)$ . Though this might sound intuitive, it’s actually a special case. If we don’t firmly grasp that it’s an exception, it can be confusing.

Suppose that a study has the point estimator $B$ for the parameter $\Theta$ . The study results are an estimate $B=b$ (typically a regression coefficient), and an estimated standard deviation² $\hat{sd}(B)=s$ .

In order to know how to combine this information with a prior over $\Theta$ in order to update our beliefs, we need to know what is the likelihood function implied by the study. The likelihood function is the probability of observing the study data $B=b$ given different values for $\Theta$ . It is formed from the probability of the observation that $B=b$ conditional on $\Theta=\theta$ , but viewed and used as a function of $\theta$ only³:

\mathcal{L}: \theta \mapsto P(B =b \mid \Theta = \theta)

The event “ $B=b$ ” is often shortened to just “ $b$ ” when the meaning is clear from context, so that the function can be more briefly written $\mathcal{L}: \theta \mapsto P(b \mid \theta)$ .

So, what is $\mathcal{L}$ ? In a typical regression context, $B$ is assumed to be approximately normally distributed around $\Theta$ , due to the central limit theorem. More precisely, $\frac{B - \Theta}{sd(B)} \sim \mathcal{N}(0,1)$ , and equivalently $B\sim \mathcal{N}(\Theta,sd(B)^2)$ .

$sd(B)$ is seldom known, and is often replaced with its estimate $s$ , allowing us to write $B\sim \mathcal{N}(\Theta,s^2)$ , where only the parameter $\Theta$ is unknown⁴.

We can plug this into the definition of the likelihood function:

\mathcal{L}: \theta \mapsto P(b\mid \theta)= \text{PDF}_{\mathcal{N}(\theta,s^2)}(b) = {\frac {1}{s\sqrt {2\pi }}}\exp \left(-{\frac {1}{2}}\left({\frac {b-\theta }{s }}\right)^{2} \right)

We could just leave it at that. $\mathcal{L}$ is the function⁵ above, and that’s all we need to compute the posterior. But a slightly different expression for $\mathcal{L}$ is possible. After factoring out the square,

\mathcal{L}: \theta \mapsto {\frac {1}{s {\sqrt {2\pi }}}}\exp \left(-{\frac {1}{2}} {\frac {(b-\theta)^2 }{s^2 }} \right),

we make use of the fact that $(b-\theta)^2 = (\theta-b)^2$ to rewrite $\mathcal{L}$ with the positions of $\theta$ and $b$ flipped:

\mathcal{L}: \theta \mapsto {\frac {1}{s {\sqrt {2\pi }}}}\exp \left(-{\frac {1}{2}}\left({\frac {\theta-b }{s }}\right)^{2} \right).

We then notice that $\mathcal{L}$ is none other than

\mathcal{L}: \theta \mapsto \text{PDF}_{\mathcal{N}(b,s^2)}(\theta)

So, for all $b$ and for all $\theta$ , $\mathcal{L}: \theta \mapsto \text{PDF}_{\mathcal{N}(\theta,s^2)}(b) = \text{PDF}_{\mathcal{N}(b,s^2)}(\theta)$ .

The key thing to realise is that this is a special case due to the fact that the functional form of the normal PDF is invariant to substituting $b$ and $\theta$ for each other. For many other distributions of $B$ , we cannot apply this procedure.

This special case is worth commenting upon because it has personally lead me astray in the past. I often encountered the case where $B$ is normally distributed, and I used the equality above without deriving it and understanding where it comes from. It just had a vaguely intuitive ring to it. I would occasionally slip into thinking it was a more general rule, which always resulted in painful confusion.

To understand the result, let us first illustrate it with a simple numerical example. Suppose we observe an Athenian man $b=200$ cm tall. For all $\theta$ , the likelihood of this observation if Athenian men’s heights followed an $\mathcal{N}(\theta,10)$ is the same number as the density of observing an Athenian $\theta$ cm tall if Athenian men’s heights followed a $\mathcal{N}(200,10)$ ⁶.

Graphical representation of $\text{PDF}_{\mathcal{N}(\theta,10)}(200) = \text{PDF}_{\mathcal{N}(200,10)}(\theta)$

When encountering this equivalence, you might, like me, sort of nod along. But puzzlement would be a more appropriate reaction. To compute the likelihood of our 200 cm Athenian under different $\Theta$ -values, we can substitute a totally different question: “assuming that $\Theta=200$ , what is the probability of seeing Athenian men of different sizes?”.

The puzzle is, I think, best resolved by viewing it as a special case, an algebraic curiosity that only applies to some distributions. Don’t even try to build an intuition for it, because it does not generalise.

To help understand this better, let’s look at at a case where the procedure cannot be applied.

Suppose for example that $B$ is binomially distributed, representing the number of successes among $n$ independent trials with success probability $\Theta$ . We’ll write $B \sim \text{Bin}(n, \theta)$ .

$B$ ’s probability mass function is

g: k \mapsto \text{PMF}_{\text{Bin}(n, \theta)}(k) = {n \choose k} \phi^k (1-\phi)^{n-k}

Meanwhile, the likelihood function for the observation of $b$ successes is

\mathcal{M}: \phi \mapsto \text{PMF}_{\text{Bin}(n, \theta)}(b) = {n \choose b} \phi^b (1-\phi)^{n-b}

To attempt to take the PMF $g$ , set its parameter $\theta$ equal to $b$ , and obtain the likelihood function would not just give incorrect values, it would be a domain error. Regardless of how we set its parameters, $g$ could never be equal to the likelihood function $\mathcal{M}$ , because $g$ is defined on $\{0,1,...,n\}$ , whereas $\mathcal{M}$ is defined on $[0,1]$ .

The likelihood function $\mathcal{Q}: P_H \mapsto P_H^2(1-P_H)$ for the binomial probability of a biased coin landing heads-up, given that we have observed $\{Heads, Heads, Tails\}$ . It is defined on $[0,1]$ . (The constant factor $3 \choose 2$ is omitted, a common practice with likelihood functions, because these constant factors have no meaning and make no difference to the posterior distribution.)

It’s hopefully now quite intuitive that the case where $B$ is normally distributed was a special case.⁷

Let’s recapitulate.

The likelihood function is the probability of $b\mid\theta$ viewed as a function of $\theta$ only. It is absolutely not a density of $\theta$ .

In the special case where $B$ is normally distributed, we have the confusing ability of being able to express this function as if it were the density of $\theta$ under a distribution that depends on $b$ .

I think it’s best to think of that ability as an algebraic coincidence, due to the functional form of the normal PDF. We should think of $\mathcal{L}$ in the case where $B$ is normally distributed as just another likelihood function.

Finally, I’d love to know if there is some way to view this special case as enlightening rather than just a confusing exception.

I believe that to say that a $\text{PDF}_{\theta,\Gamma}(b)=\text{PDF}_{b,\Gamma}(\theta)$ (where $\text{PDF}_{\psi,\Gamma}$ denotes the PDF of a distribution with one parameter $\psi$ that we wish to single out and a vector $\Gamma$ of other parameters), is equivalent to saying that the PDF is symmetric around its singled-out parameter. For example, a $\mathcal{N}(\mu,\sigma^2)$ is symmetric around its parameter $\mu$ . But this hasn’t seemed insightful to me. Please write to me if you know an answer to this.

Thanks to Gavin Leech and Ben West for feedback on a previous versions of this post. ↩
I do not use the confusing term ‘standard error’, which I believe should mean $sd(B)$ but is often also used to also denote its estimate $s$ . ↩
I use uppercase letters $\Theta$ and $B$ to denote random variables, and lower case $\theta$ and $b$ for particular values (realizations) these random variables could take. ↩
A more sophisticated approach would be to let $sd(B)$ be another unknown parameter over which we form a prior; we would then update our beliefs jointly about $\Theta$ and $sd(B)$ . See for example Bolstad & Curran (2016), Chapter 17, “Bayesian Inference for Normal with Unknown Mean and Variance”. ↩
I don’t like the term “likelihood distribution”, I prefer “likelihood function”. In formal parlance, mathematical distributions are a generalization of functions, so it’s arguably technically correct to call any likelihood function a likelihood distribution. But in many contexts, “distribution” is merely used as short for “probability distribution”. So “likelihood distribution” runs the risk of making us think of “likelihood probability distribution” – but the likelihood function is not generally a probability distribution. ↩
We are here ignoring any inadequacies of the $B\sim N(\Theta,s^2)$ assumption, including but not limited to the fact that one cannot observe men with negative heights. ↩
Another simple reminder that the procedure couldn’t possibly work in general is that in general the likelihood function is not even a PDF at all. For example, a broken thermometer that always gives the temperature as 20 degrees has $P(B=20 \mid \theta) = 1$ for all $\theta$ , which evidently does not integrate to 1 over all values of $\theta$ .

To take a different tack, the fact that the likelihood function is invariant to reparametrization also illustrates that it is not a probability density of $\theta$ (thanks to Gavin Leech for the link). ↩

How to circumvent Sci-Hub ISP block

2021-05-15T00:00:00+00:00

In the UK, many internet service providers (ISPs) block Sci-Hub. However, a simple proxy is enough to circumvent this (you don’t even need a VPN). Routing requests through a suitable¹ proxy lets you open Sci-Hub in your regular browser as if it weren’t blocked².

Routing all your traffic through a proxy may come with privacy and security concerns, and will slow your connection a bit. We want to use our proxy only for accessing Sci-Hub.

You can use extensions like ProxySwitchy to tell your browser to automatically use certain proxies, or no proxy at all, for sets of websites that you define.

Unfortunately, this extension, and others like it, require permissions to insert arbitrary JavaScript into any page you visit (the web store accurately explains that the extension can “read and change all your data on the websites you visit”). That’s likely due to insufficiently granular permission definitions by Chrome, and is not the fault of the presumably well-intentioned extension authors. But it freaks me out a little bit (bad things have happened).

Luckily, we can achieve the same effect by writing our own proxy auto-configuration file. A proxy auto-configuration or PAC file contains just a single JavaScript function like this:

function FindProxyForURL (url, host) {
  
  // Sci-Hub requests
  if (shExpMatch(host, 'sci-hub.se') || shExpMatch(host,'*.sci-hub.se')) {

                  // Your proxy address and port number
    return 'PROXY 123.456.789:9279'; 
  }

  // All other requests
  return 'DIRECT';
}

We can instruct the operating system to read this file. Search for instructions on Google (example).

When you use the proxy to access Sci-Hub for the first time in a browser session, the browser will ask you for the username and password to your proxy server. If you’re using Chrome, I’d recommend saving the credentials into the browser’s password manager to avoid having to enter them again.³

There are many free proxies on the Internet, but I find that using the services of an actual for-profit proxy company is well worth it, for the greater speed and reliability. Currently webshare.io (referral link) offers 1 GB per month free, which is quite a lot of Sci-Hub PDFs. After that you can get 250 GB for $2.99 per month.⁴

Step by step instructions

Create an account on webshare.io (referral link)
Choose a proxy from your list, and copy its address and port number into your PAC file, following the pattern above.
Set your operating system to read its proxy settings from this PAC file⁵. Instructions for this are easy to Google (example).
Open Sci-Hub in your browser. Enter your proxy username and password and optionally save these credentials in the browser. You can find the credentials in your webshare.io account.
Don’t forget to only use Sci-Hub to look at really old papers that have lapsed into the public domain :)

Obviously, the proxy must not itself be on a network that blocks Sci-Hub. I have not come across any proxy that blocks Sci-Hub in this way. ↩
Changing your DNS resolver to a public one like Google’s instead of your ISP’s is not sufficient as of 2021, for two ISPs I’ve tested, and I suspect for all UK ISPs that implement blocking. (Many people believe changing the DNS resolver is sufficient. Probably ISPs used to implement simple DNS level blocking and have recently upped their game.) My guess is that instead of merely blocking the request to resolve sci-hub.se at the DNS resolver level, the ISPs are also doing a reverse lookup on every requested IP address to check whether it corresponds to a blacklisted domain. ↩
You want to use the Chrome password manager because third-party password managers such as 1Password are not able to auto-fill credentials when logging in to a proxy server (as opposed to logging into a webpage). Note that if you have a third-party password manager extension installed this will disable the browser setting “Offer to save passwords”. I recommend that you temporarily disable your password manager extension, log in to your proxy server, save the password into Chrome, and then enable the extension again. ↩
Their home page exemplifies a dark pattern by not showing the pricing by the GB; it just says you get ‘up to unlimited’ bandwidth. You’ll be able to see the actual pricing after you create an account. ↩
One gotcha is that Windows 10 forces you to call your PAC file from a web server; it cannot be a local file (??!). To work around this, you can upload your file as a Gist and link to the /raw. ↩

bayes.net

Shut up and SWE-bench

Method

Results

MirrorCode and making evals that are hard, but actually fair

Avoiding absurd reverse-engineering

Telling AIs what’s in scope

btw, don’t underestimate these tasks

Security mindset for evals

It’s open source

A few inference scaling curves

Claude 4 hacked SWE-bench by peeking at future commits

How to run SWE-bench Verified in one hour on one machine

Background

SWE-bench and Docker

Docker layering

Anatomy of a SWE-bench Dockerfile

Moving the git clone operation

Should the git history be included?

The matplotlib 1.9 GB top layer

Disabling the pip cache (or how to go insane)

Impact on size

Running SWE-bench Verified in about an hour

How to use our image registry

How should we analyse survey forecasts of AI timelines?

Headline results

See previous

Compare mine (3/4 suggestions) with previous

Suggested textual description

Contents

The distribution of raw responses

Example: Retail Salesperson occupation

Timing of human-level performance

Full Automation of Labor (FAOL)

Box Plot

Fixed Probabilities

Fixed Years

High Level Machine Intelligence (HLMI)

Box Plot

Fixed Probabilities

Fixed Years

Aggregation

Possible methods

Mean vs Median aggregation

Full automation of labor (FAOL)

High Level Machine Intelligence (HLMI)

Truck Driver

Surgeon

Retail Salesperson

AI Researcher

Why the mean and median differ

Aside: the winsorized geometric mean of odds

Distribution fitting

Why fit a distribution?

Limitations of previous analyses

Constraints of Gamma distribution

Inappropriate loss function

Flexible distributions

Elevated computing

Details

Previous versions of this idea

How to run Cronicle (a cron replacement) in a Docker container

How much of the fall in fertility could be explained by lower mortality?

Constant number of children

England (1861-1951)

Sub-Saharan Africa (1990-2017)

A more sophisticated model of fertility decisions

Extensions of Barro-Becker model

The special case of the normal likelihood function

How to circumvent Sci-Hub ISP block

Step by step instructions