DoubleAI: an agent redesigns a network during training; DeepSeek-V4.1-Flash raising AIME/HMMT 2026 from 64% to 71% with a self-written code layer

Can LLMs redesign themselves as they train?

URL: https://www.doubleai.com/research/can-llms-redesign-themselves-as-they-train Published: Nov 6, 2026 Type: Blog Topics: Publications

Architecture is the last hand-designed piece. We make it part of training.

TL;DR

We put an AI agent inside a neural network's training run, where it redesigns the network on the fly.

The agent does what we call mechanistic intervention. It reads the network's internal state, much as mechanistic interpretability research does, but instead of explaining a finished network, it changes the network while it is still learning. The agent diagnoses what the network fails to learn and fixes the problem by changing the data, the objective, the update rule or the architecture itself.

This requires an entirely new toolkit, which the agent builds itself. On nanochat, the agent discovered one of its main instruments, which we call the phantom gradient: a measure of unmet demands.

The agent writes small programs and grows them into the network as code layers. The code layers are often “alien”. For example, one such layer computes gradients of hidden representations within a single forward pass and uses those to self-correct.

On CLRS-Text, a dataset of classical algorithms, the agent takes on jaggedness. We find that a transformer trained on this dataset handles the input sizes it was trained on, yet fails on the sizes in between, even when trained for twice as long. Code layers grown during training raise accuracy at those in-between sizes from 33% to 79%.

A model can also do this to itself. We let DeepSeek-V4.1-Flash act as the agent on itself. Using a code layer that reads its own hidden states, it raised its accuracy on AIME 2026 and HMMT 2026 from 64% to 71%.

Today, a transformer still transforms everything but itself. We are on the cusp of models that train and re-architect themselves, with people setting only the goal and the rules. Models will be able to carry themselves beyond the tasks they already know: they'll do science on themselves, testing ideas, experimenting and expanding their capabilities.