HEXAGONbeta

Do Self-Improving Coding Agents Beat Textbook Baselines? Evidence from a Darwin Gödel Machine Port and Two Published Systems

A mix of human-written and AI-generated textHuman understanding: all partscs.AI — Artificial Intelligence

Contributed by Tejasvi Singh Tomar ↗

Version 1 / Oct 28, 2026 / CC BY 4.0

Abstract

Self-improving agent harnesses such as the Darwin Gödel Machine (DGM) let a coding agent edit its own scaffolding, and they keep the edits that score better and report large gains on coding benchmarks. In three pre-registered rounds, we compare what such loops find with standard engineering changes ("textbook moves"), one-shot redesigns by a language model, and placebo loops that make no edits; we also re-run released agents of the DGM and of ADAS with newer models. On hard LiveCodeBench problems with Claude Haiku 5.5, the starting agent often stops on an empty turn because its output-token limit is too small for the model's reasoning. A single textbook move, which raises this limit to 16,000 tokens and makes three other changes to the model loop, recovers about 90% of the gain of the best loop agent and costs less than half as much per task. Four of five one-shot redesigns by Claude Opus 5.5 also raise the limit, to 8,192 tokens, without evaluation feedback. The best loop agent still leads the textbook moves by 1–5 percentage points in pass rate. On held-out SWE-bench Verified tasks, run in the DGM's harness, the DGM's released best agent does not outperform its initial agent; three textbook moves on the initial agent outperform the best agent at a quarter of its cost, whether or not the patches of runs that ended early, most of them the best agent's, are kept. Claims of self-improvement should be measured against textbook moves, one-shot redesigns, placebo loops, held-out tasks and newer models.

References

  1. Aleithan, Reem; Xue, Haoran; Mohajer, Mohammad Mahdi; Nnorom, Elijah; Uddin, Gias; Wang, Song. SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv preprint arXiv:2410.06992. 2024
  2. Brown, Bradley; Juravsky, Jordan; Ehrlich, Ryan; Clark, Ronald; Le, Quoc V.; Ré, Christopher; Mirhoseini, Azalia. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv preprint arXiv:2407.21787. 2024
  3. Cawley, Gavin C.; Talbot, Nicola L. C. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research, vol. 11, pp. 2079–2107. 2010
  4. Chen, Zihan; Zhu, Di; Zheng, Lei Nico. Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims. arXiv preprint arXiv:2610.27610. 2026
  5. Chowdhury, Neil; Aung, James; Chan, Jun Shern; Jaffe, Oliver; Sherburn, Dane; Starace, Giulio; Mays, Evan; Dias, Rachel; Aljubeh, Marwan; Glaese, Mia; Jimenez, Carlos E.; Yang, John; Ho, Leyton; Patwardhan, Tejal; Liu, Kevin; Madry, Aleksander. Introducing SWE-bench Verified. 2024. Published 13 August 2024, updated 24 February 2025. OpenAI blog post, https://openai.com/index/introducing-swe-bench-verified/link
  6. Ding, Tianyu; Tong, Yaocheng; Yang, Muzhi; Jiang, Fei; Yang, Lei; Yu, Boya. Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk. arXiv preprint arXiv:2610.14460. 2026
  7. Ding, Wenxuan; Lu, Qin; Yu, Changlong; Li, Sha; Jin, Shuowei; Liu, Xin; Shi, Zhenyu; Durrett, Greg. Harness-Delta Attribution: Dissecting Gains From Harness Evolution. arXiv preprint arXiv:2610.22466. 2026
  8. Gabor, Jonathan; Lynch, Jayson; Rosenfeld, Jonathan. EvilGenie: A Reward Hacking Benchmark. arXiv preprint arXiv:2511.21654. 2025
  9. He, Rong. Does Failure Feedback Direct Agent Self-Modification?. arXiv preprint arXiv:2610.14662. 2026
  10. Henderson, Peter; Islam, Riashat; Bachman, Philip; Pineau, Joelle; Precup, Doina; Meger, David. Deep Reinforcement Learning That Matters. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1. 2018DOI
Show all 41 referencesShow fewer references
  1. Hu, Litao; Tang, Yutong. The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules. arXiv preprint arXiv:2610.09239. 2026
  2. Hu, Shengran; Lu, Cong; Clune, Jeff. Automated Design of Agentic Systems. International Conference on Learning Representations. 2025
  3. Jain, Naman; Han, King; Gu, Alex; Li, Wen-Ding; Yan, Fanjia; Zhang, Tianjun; Wang, Sida; Solar-Lezama, Armando; Sen, Koushik; Stoica, Ion. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. International Conference on Learning Representations, pp. 58791–58831. 2025
  4. Jimenez, Carlos E.; Yang, John; Wettig, Alexander; Yao, Shunyu; Pei, Kexin; Press, Ofir; Narasimhan, Karthik. SWE-bench: Can Language Models Resolve Real-world Github Issues?. International Conference on Learning Representations, pp. 54107–54157. 2024link
  5. Jwalapuram, Prathyusha; Lin, Hehai; Li, Chuyuan; Jiao, Fangkai; Wang, Sudong; Ming, Yifei; Ke, Zixuan; Qin, Chengwei; Carenini, Giuseppe; Joty, Shafiq. The Illusion of Multi-Agent Advantage. arXiv preprint arXiv:2606.13003. 2026
  6. Kapoor, Sayash; Stroebl, Benedikt; Siegel, Zachary S.; Nadgir, Nitya; Narayanan, Arvind. AI Agents That Matter. arXiv preprint arXiv:2407.01502. 2024
  7. Li, Liam; Talwalkar, Ameet. Random Search and Reproducibility for Neural Architecture Search. Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, vol. 115, pp. 367–377. 2020link
  8. Liang, Yunhao; Gan, Chengguang; Ying, Ruixuan; Wei, Hanjun; Cui, Zhe; Ni, Shiwen. Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem. arXiv preprint arXiv:2608.19626. 2026
  9. Lin, Jiahang; Liu, Shichun; Pan, Chengjun; Lin, Lizhi; Dou, Shihan; Xi, Zhiheng; Huang, Xuanjing; Yan, Hang; Han, Zhenhua; Gui, Tao; Jiang, Yu-Gang. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. arXiv preprint arXiv:2604.25850. 2026
  10. Lu, Xinyu; Wang, Tianshu; Wang, Pengbo; Wen, Zujie; Zhang, Zhiqiang; Zhou, Jun; Cao, Boxi; Lu, Yaojie; Lin, Hongyu; Han, Xianpei; Sun, Le. The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?. arXiv preprint arXiv:2606.04455. 2026
  11. Ludwig, Nikolai; Ahmad, Wasi Uddin; Majumdar, Somshubra; Ginsburg, Boris. Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks. arXiv preprint arXiv:2609.06780. 2026
  12. Oved, Tal; Pony, Roi; Naparstek, Oshri; Barzelay, Udi. LLM Evolutionary Search Strategies Should be Compared by Tokens. arXiv preprint arXiv:2610.23977. 2026
  13. Robeyns, Maxime; Szummer, Martin; Aitchison, Laurence. A Self-Improving Coding Agent. arXiv preprint arXiv:2504.15228. 2025
  14. Srikanth, Dhruv; Zhao, Bingchen; Xu, Dixing; Wu, Yuxiang; Jiang, Zhengyao. Recursive self-improvement of AI research agents. arXiv preprint arXiv:2609.26457. 2026
  15. SWE-bench GitHub issue #465. Repo State Loopholes During Agentic Evaluation. 2025. Opened 3 September 2025. https://github.com/SWE-bench/SWE-bench/issues/465link
  16. Tian, Yuan; Hu, Bing; Wang, Hao; Lu, Binghang; Wu, Fang. Harness Evolution Hits a Ceiling: When Weight Training Should Begin. arXiv preprint arXiv:2610.11655. 2026
  17. Verma, Yuvraj. Try Again, Don't Look Back: Blind Resampling Outperforms Self-Repair in Small Code Models. arXiv preprint arXiv:2607.26117. 2026
  18. Wang, Wenyi; Piękos, Piotr; Nanbo, Li; Laakom, Firas; Chen, Yimeng; Ostaszewski, Mateusz; Zhuge, Mingchen; Schmidhuber, Jürgen. Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine. International Conference on Learning Representations, pp. 80356–80386. 2026
  19. Wang, Xuezhi; Wei, Jason; Schuurmans, Dale; Le, Quoc V.; Chi, Ed H.; Narang, Sharan; Chowdhery, Aakanksha; Zhou, Denny. Self-Consistency Improves Chain of Thought Reasoning in Language Models. International Conference on Learning Representations. 2023link
  20. Wang, Yike; Zhu, Huaisheng; Hu, Zhengyu; Yuan, Yige; Chen, Zhengyu; Senthil, Shakti; Hajishirzi, Hannaneh; Tsvetkov, Yulia; Smith, Noah A.; Dasigi, Pradeep; Xiao, Teng. Rethinking the Evaluation of Harness Evolution for Agents. arXiv preprint arXiv:2607.12227. 2026
  21. Wang, You; Pradel, Michael; Liu, Zhongxin. Are ``Solved Issues'' in SWE-bench Really Solved Correctly? An Empirical Study. arXiv preprint arXiv:2503.15223. 2025
  22. Xu, Bowen; Chen, Boyu. What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects. arXiv preprint arXiv:2610.05533. 2026
  23. Xue, Hui; Yang, Fan. Rethinking Self-Evolving Agents: Do We Still Need Prescribed Optimization Pipelines?. arXiv preprint arXiv:2608.09629. 2026
  24. Yang, Antoine; Esperança, Pedro M.; Carlucci, Fabio M. NAS evaluation is frustratingly hard. International Conference on Learning Representations. 2020link
  25. Yin, Xunjian; Wang, Xinyi; Pan, Liangming; Lin, Li; Wan, Xiaojun; Wang, William Yang. Gödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 27890–27913. 2025DOI
  26. Zelikman, Eric; Lorch, Eliana; Mackey, Lester; Kalai, Adam Tauman. Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation. arXiv preprint arXiv:2310.02304. 2023
  27. Zhang, Jenny; Hu, Shengran; Lu, Cong; Lange, Robert; Clune, Jeff. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. International Conference on Learning Representations, pp. 104223–104294. 2026
  28. Zhang, Jenny; Zhao, Bingchen; Yang, Wannan; Foerster, Jakob; Clune, Jeff; Jiang, Minqi; Devlin, Sam; Shavrina, Tatiana. Hyperagents. arXiv preprint arXiv:2603.19461. 2026
  29. Zhang, Jiayi; Xiang, Jinyu; Yu, Zhaoyang; Teng, Fengwei; Chen, Xionghui; Chen, Jiaqi; Zhuge, Mingchen; Cheng, Xin; Hong, Sirui; Wang, Jinlin; Zheng, Bingnan; Liu, Bang; Luo, Yuyu; Wu, Chenglin. AFlow: Automating Agentic Workflow Generation. International Conference on Learning Representations, pp. 34040–34077. 2025
  30. Zhong, Ziqian; Raghunathan, Aditi; Carlini, Nicholas. ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases. International Conference on Learning Representations, pp. 124959–124986. 2026
  31. Zhu, Yuxuan; Jin, Tengjun; Pruksachatkun, Yada; Zhang, Andy; Liu, Shu; Cui, Sasha; Kapoor, Sayash; Longpre, Shayne; Meng, Kevin; Weiss, Rebecca; Barez, Fazl; Gupta, Rahul; Dhamala, Jwala; Merizian, Jacob; Giulianelli, Mario; Coppock, Harry; Ududec, Cozmin; Sekhon, Jasjeet; Steinhardt, Jacob; Kellermann, Antony; Schwettmann, Sarah; Zaharia, Matei; Stoica, Ion; Liang, Percy; Kang, Daniel. Establishing Best Practices for Building Rigorous Agentic Benchmarks. arXiv preprint arXiv:2507.02825. 2025

Version history

  1. v1Submitted by Tejasvi Singh TomarInitial depositCurrentOct 28, 2026