\section{Published systems}\label{sec:published}
\subsection{The DGM's SWE-bench agents}
\begin{table}[t]\centering\small
\caption{The DGM's released SWE-bench agents and the initial agent with textbook moves, run through the DGM's own
harness with Claude Sonnet~4.5 on SWE-bench Verified tasks whose reference patch passes today: fresh tasks outside the
DGM's evaluation set, and the DGM's selection tasks. One run per task. Early: runs, of all 293, in which the agent
process ended before the harness recorded a patch; they count as unresolved.}\label{tab:swe}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{lrrrr}\toprule
agent & fresh & DGM selection & \$/run & early \\ \midrule
DGM initial agent & \vsweInitFreshN/\vsweInitFreshOf\ (\vsweInitFresh\%) & \vsweInitSelN/\vsweInitSelOf & \vsweInitUsd & \vsweInitEarly \\
DGM reported best agent & \vsweBestFreshN/\vsweBestFreshOf\ (\vsweBestFresh\%) & \vsweBestSelN/\vsweBestSelOf & \vsweBestUsd & \vsweBestEarly \\
initial + textbook moves & \vsweMovesFreshN/\vsweMovesFreshOf\ (\vsweMovesFresh\%) & \vsweMovesSelN/\vsweMovesSelOf & \vsweMovesUsd & \vsweMovesEarly \\
\quad + regression-test retries & \vsweRetryFreshN/\vsweRetryFreshOf\ (\vsweRetryFresh\%) & \vsweRetrySelN/\vsweRetrySelOf & \vsweRetryUsd & \vsweRetryEarly \\ \bottomrule
\end{tabular}\end{table}
The DGM reports that its search took the initial agent from 20\% to 50\% on its 200 SWE-bench tasks with Claude~3.5
Sonnet, which is no longer served. We ran all four agents with Claude Sonnet~4.5 on Verified tasks outside the DGM's
200-task evaluation set (Table~\ref{tab:swe}); Sonnet~4.5 is newer than the model the DGM used. As run in the DGM's harness, the reported best agent resolves $\vsweBestMinusInitAbs$ points
[$\vsweBestMinusInitAbslo$, $\vsweBestMinusInitAbshi$] fewer fresh tasks than the initial agent it came from, and the
initial agent with three textbook moves (defined in \S\ref{sec:round2}) resolves $\vsweMovesMinusBestAbs$ points [$\vsweMovesMinusBestAbslo$, $\vsweMovesMinusBestAbshi$] more
than the best agent, at about a quarter of its cost per task. The textbook moves alone add $\vsweMovesMinusInitAbs$ points
[$\vsweMovesMinusInitAbslo$, $\vsweMovesMinusInitAbshi$] to the initial agent at the same cost. On the DGM's own selection
tasks the textbook agent leads the best agent by $\vsweMovesMinusBestSelAbs$ points [$\vsweMovesMinusBestSelAbslo$,
$\vsweMovesMinusBestSelAbshi$].

Most of the best agent's deficit relative to its initial agent comes from runs that end early. In \vsweBestEarly{} of
its \vsweBestRuns{} runs the agent process ended before the harness recorded a patch, and the DGM's harness then discards the
run, as it does for every agent. We could not determine the cause for this model; the exit codes, a scan of the saved
logs and two pilots are described in Appendix~\ref{app:swe}. The best agent's design (up to three attempts, each checked
with the repository's regression tests, plus summarization of long conversations) produces long sessions: its median
run took \vsweBestMedMin{} minutes, compared with \vsweMovesMedMin{} for the textbook agent. Among the runs that finished, the best
agent resolves the largest share of fresh tasks ($\vsweBestFinFresh\%$; the textbook
agent resolves $\vsweMovesFinFresh\%$), but this comparison conditions on finishing and favours agents that end early on hard tasks.

In an exploratory sensitivity analysis, we graded the patch that each early-ended run left in the repository, as if
the harness had kept it. This does not show what a completed run would have produced. Of the best agent's
\vsweBestEarly{} early-ended runs, \vsensBestPatched{} had a patch, and \vsensBestPatchedResolved{} of those resolve their
task. With these patches kept, the best agent resolves $\vsensBestFresh\%$ of fresh tasks. That is $\vsensBestMinusInit$
points [$\vsensBestMinusInitlo$, $\vsensBestMinusInithi$] relative to the initial agent, and the textbook agent
($\vsensMovesFresh\%$) still leads it by $\vsensMovesMinusBestAbs$ points [$\vsensMovesMinusBestAbslo$,
$\vsensMovesMinusBestAbshi$]. On the DGM's selection tasks, with patches kept, the textbook agent's lead shrinks to
$\vsensMovesMinusBestSelAbs$ points [$\vsensMovesMinusBestSelAbslo$, $\vsensMovesMinusBestSelAbshi$], and the best agent
is $\vsensBestMinusInitSel$ [$\vsensBestMinusInitSello$, $\vsensBestMinusInitSelhi$] relative to its initial agent. Under
this newer model, then, neither reading of the fresh tasks shows a significant gain for the released best agent over
the initial agent it came from, and the textbook agent remains ahead of it. The advantage the DGM reported (from 20\% to 50\%) does not carry over.

\input{sec_swe_equal_v3.tex}

\paragraph{Harness findings.} The DGM's harness runs the task's evaluation script inside the agent's container before
the agent starts, to install the environment. That script applies the hidden tests, runs them and restores only the test
files it modified. Afterwards a pytest cache directory remained in all 106 pytest-based tasks we ran (of 293; the django and sympy
tasks use their own test runners); in the
three we inspected, its \texttt{lastfailed} file named the failing hidden tests: in two it listed exactly the test that
the reference patch must make pass, and in the third those 4 tests among 142.
Test files that the hidden patch adds also stay in the repository (8 of 300 fresh and 7 of the DGM's 200 tasks). To our knowledge this has not been reported; the nearest known case is agents reading
future commits from the repository's history \citep{swebench2025issue465}. None of the DGM's released chats reads the
cache, so its reported numbers do not appear to use it, but a self-modifying agent could learn to. All our runs remove
this residue after setup. Separately, 59 of the 352 tasks we built no longer pass with their reference patch under
today's package versions and were excluded for every agent.

\subsection{ADAS}
\begin{table}[t]\centering\small
\caption{ADAS's best-by-validation discovered agents (found with gpt-3.5) against chain-of-thought self-consistency
with a matched number of samples (one extra aggregation call on DROP), both run with Claude Haiku~5.5 on ADAS's test
sets. Calls per question in parentheses. Last column: the gain over CoT-SC that ADAS's archive reports with gpt-3.5 (for
DROP computed from archive medians that lie above their own reported intervals).}\label{tab:adas}
\footnotesize\setlength{\tabcolsep}{3pt}
\begin{tabular}{lrrrrrr}\toprule
domain & ADAS agent & CoT-SC & CoT & agent $-$ CoT-SC [95\% CI] & agent $-$ CoT & archive gain \\ \midrule
\csname @@input\endcsname tab_adas.tex
\bottomrule\end{tabular}\end{table}
Run with a current model, ADAS's discovered agents lose to self-consistency with matched calls on MGSM and DROP, tie on
MMLU and win on GPQA (Table~\ref{tab:adas}); a single chain-of-thought call matches or beats them on three of four
domains with a quarter to a tenth of the calls. On DROP the discovered agent gives longer answers, which ADAS's own F1 metric penalizes: on the items where it
scores below CoT-SC, its answer is longer in \vdropWorseLonger{} of \vdropWorse{} (median \vdropLenAgent{} characters,
compared with \vdropLenCot{}). The agents with the best test score in the archive behave the same way (on MGSM it is the same agent). This agrees with
\citet{jwalapuram2026illusion}, who find automatically designed multi-agent systems behind CoT-SC with current models.