\section{Discussion}\label{sec:discussion}
\input{sec_hyp_v2.tex}
\input{sec_hyp3_v3.tex}
Tables~\ref{tab:hyp} and~\ref{tab:hyp3} list the registered hypotheses and their outcomes, and the findings in the introduction
summarize them together with the exploratory comparisons. Read together, they show that the loops mostly
found standard engineering, which a one-shot redesign also reaches at a lower cost per task whether or not it sees
failure logs; they do not show that loops find nothing.

Two readings follow. The first concerns what search contributes. In most of our settings, a baseline that uses no
search (textbook moves, or a single redesign from the same failure logs) reached the level of the loops' best agents,
or came within a few points of it, at lower cost. The exceptions run both ways: on gpt-oss-120b an agent found by a loop
on another model gained more than our textbook set did, in a comparison across batches, and ADAS's discovered agent wins on GPQA.
Diagnosis also carries information: once the request list contains nothing aimed at the empty-turn failure,
random-diagnosis loops gain much less than DGM loops (\S\ref{sec:random-clean}). The loops' costs are
many evaluations and inflated selection scores (\S\ref{sec:round1}). The second reading concerns transfer to a
new task model. A harness is tuned to the model it was built for, whether by a loop or directly: the engineered seed
loses to the DGM seed under gpt-oss-120b, and it is ahead of loop 3's agent under Haiku~4.5 and Sonnet~4.6, about
level under Sonnet~5.5, and behind it under gpt-oss-120b and Gemini~2.5 Flash (Appendix~\ref{app:round1}). Published
self-improvement results should therefore be re-measured when the task model changes.

\paragraph{Recommendations.} Claims of self-improvement should report (i) gains on tasks that no selection step saw,
(ii) a placebo loop or its order-statistics prediction, (iii) the seed resampled at the agent's cost, (iv) the seed
with textbook moves and a one-shot redesign from the same failure logs, measured in the same batch as the loop's agent
(our own registered tests did not fully meet this), and (v) the released agent re-run with a newer model. Harnesses
should keep hidden tests out of the agent's container, keep the agent's work when its process ends early, and check that
each task's reference patch still passes.

\paragraph{Limitations.} Loops were run on one task family with one to nine loops per arm, and the confidence intervals
over problems hold the realized loops fixed. At the level of loops, differences between loop types are significant only
for the clean random-diagnosis loops of the third round (\S\ref{sec:random-clean}). The SWE-bench and other-model evaluations use one run per task, and budget and time limits censor several
agents under other models (Appendix~\ref{app:cens}). Claude~3.5 Sonnet, the DGM's original model, is no longer served,
so we cannot separate the effect of the model change from a failure to replicate. We could not determine what ends the
best agent's runs early under Claude Sonnet~4.5; machine load and memory do not explain it (\S\ref{sec:swe-equal}). The SWE-bench batch at equal output budget used \wN{} tasks. HGM's released agent and the DGM's Polyglot agents were not run. The proxy caps output at 16{,}000 tokens for
every arm. The engineered seed, the source of the textbook moves, was written while pilot runs that included \vnpN{}
of the fresh problems took place, and their logs showed the DGM seed's empty responses, so the textbook moves are not
independent of what failure logs show; without those problems the main numbers barely change (Appendix~\ref{app:pilot}).
The request list of the second-round random-diagnosis loops was drafted after the first round had exposed the
empty-turn failure, and three of its requests aim at or near it (\S\ref{sec:round2}); the third round repeats the test
without them (\S\ref{sec:random-clean}). Several
comparisons were added after earlier results (Appendix~\ref{app:dev}, items 3 and 15); those of item 3 were specified
before they were computed, and those of item 15 after they were computed. Under Haiku~4.5 most runs of the DGM seed end with its agent process killed by a signal, which the engineered
seed avoids (Appendix~\ref{app:othermodels}). Heavy load on the shared machine affected part of one exploratory batch
and the last generations of three loops, and the second-round SWE-bench run used an oversubscribed machine
(Appendix~\ref{app:dev}).