GPT-6 Astra lifted a tiny Qwen model’s math score sixfold by editing its weights

Researchers at MIT CSAIL report that OpenAI’s GPT-6 Astra raised the math score of the small open-weight model Qwen3.5-0.8B from 7.9% to 48.1% on MATH-500 by directly editing about 0.016% of its weights, with no gradient-based training. The work, which spread first on Reddit, is described in an authors’ blog post dated November 2, 2026.

One direction subtracted from two layers

The post, by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, calls the approach “Weight-Space Programming”: a general-purpose language model reads and writes another network’s weights directly, without an optimizer. Astra had a Python environment and could run forward passes on practice problems from GSM8K and MATH, but no training and no gradients.

According to the authors, Astra traced the model’s looping behavior to a single direction written by the MLPs in layers 15 and 19. It then subtracted that direction from 64 output columns in each layer. That is 131,072 weights out of roughly 0.8 billion.

The results depended heavily on how much Astra was allowed to test its work:

• Base model: 7.9% on MATH-500, 13.8% on GSM8K, 21.0% on IFEval, with 93.0% of MATH-500 answers hitting the 32,768-token limit.
• After 10 forward passes: 12.3%, 25.7% and 49.0%, with 87.2% of answers hitting the limit.
• After 25 forward passes: 14.1%, 25.1% and 39.9%, with 86.4% hitting the limit.
• After 50 forward passes: 48.1%, 67.1% and 62.3%, with 33.1% hitting the limit.

The authors also report experiments outside language. According to a VentureBeat summary of the post, Astra matched four game-playing networks (Breakout, Pong, Pac-Man, Freeway) to their games, patched a stuck simulated driving controller, and built game networks from written rules. Logs are published on GitHub, and the team says it is building a benchmark and harness, with no release date given.

The gain is a repair, not new math skill

The headline number invites the reading that Astra taught the model arithmetic. The authors say otherwise. The share of MATH-500 answers that ran to the token limit fell from 93.0% to 33.1%, which points to the edit stopping a repetition loop so the model could finish answers it was already capable of. Their own negative result supports this. Astra could not improve Gemma 3 270M, which rarely loops. In the authors’ words: “Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.”

The post lists other limits. Astra could not tell a sycophantic model from a correct one using weights alone, and succeeded only after it was allowed to inspect attention patterns on texts it had written. Some benchmarks used very few samples, such as one on MMLU-Pro and two each on GSM8K and IFEval. The authors describe current capability as “still limited”, mainly by context length and harness inefficiencies.

Older editing tricks, now chosen by a model

Targeted weight editing is not new. The post points to ROME (NeurIPS 2022) and MEMIT (ICLR 2023), which changed specific facts stored in a model’s weights. What the MIT team claims is different: a general-purpose model, given only practice problems and tool access, decides what to change and where, and does so for a failure that is behavioral rather than a single fact.

That is hard because weights are not labeled. Finding one repetition-related direction among roughly 0.8 billion numbers means forming hypotheses, testing them on the live model and revising, which is why the number of forward passes mattered so much here. The effort was also not small: per the account VentureBeat summarized, the Qwen run took 50 evaluation requests, 158 conversational turns, 80 Python runs and 20.3 hours, including 4.7 hours lost to a service outage.

Outside voices are split. Yuchen Jin of Databricks, who is not part of the work, told VentureBeat’s readers via the report that dependable weight editing could cut some maintenance and tuning costs, though no savings have been shown. PauseAI’s chief executive, Maxime Fournes, called it “a terribly reckless research direction” on X. The study does not establish any advantage over conventional fine-tuning in cost, speed or effectiveness, and it does not show recursive self-improvement, since gains were not shown to compound across cycles.

What readers can and cannot use yet

Nothing here is a product. Qwen3.5-0.8B is available under Apache 2.0, and the logs are public, but the harness and benchmark are not. Access to GPT-6 Astra for this kind of work is not described in the sources we read. The results are the authors’ own, run on a 0.8-billion-parameter model, and independent reproduction has not been reported. Developers who tune small models for reasoning should treat it as a pointer: a model that loops can score far below its real ability, and a precise fix to a few layers can recover some of it. It is not evidence that larger models can be improved this way. Related work on automated model changes appears in our coverage of Google’s Harness2 and the Heretic 2.0 model-modification tool.