Qwen3.8-27B jumps from 39% to 55% on Terminal-Bench with a small LoRA adapter
Hugging Face trained one LoRA per agent harness for Qwen3.8-27B, then merged them. Its own tests show higher scores and far fewer tool calls, though some figures do not match.

Hugging Face says that a small LoRA adapter, trained with reinforcement learning inside an agent harness, lets Qwen3.8-27B score 55% pass@1 on Terminal-Bench 2.1 at about 1.5K tokens per task. The same model with the pi harness alone scores 39% at 2.9K, according to the company’s post.
The team trained one adapter for each of three open harnesses (pi, Hermes Agent and OpenCode) and then merged the three into a single “super adapter”. The post summarizes the result in one sentence: “Each harness adapter beats the base model under its own harness on both benchmarks, for both performance and efficiency.”

How the adapters were trained and what was released
The harnesses run exactly as they ship. A proxy sits between each harness and vLLM and records the tokens the trainer learns from, so no one rewrites the agent loop. Training used TRL’s AsyncGRPOTrainer on Hugging Face Jobs, with one B200 for vLLM and one for training per adapter. The final learning rate was 2e-5, and the reward combined verifier correctness with a 0.1 tool-efficiency bonus.
The tasks come from three sources: bugs injected into real repositories (SWE-smith, Python and Go), real GitHub issues (SWE-Gym) and tasks generated from merged pull requests (Tasksmith). Of 1,127 Hub candidates, 858 passed validation. Hugging Face then ran the base model on each task and kept the 171 it solved on some attempts but not all. The released dataset, harness-hub/agentic-coding-envs, holds 871 tasks from 35 repositories.

The three adapters were trained for 598 cumulative steps, on 78,114 samples and 167M tokens. A plugin called safeagents checked 1.3M tool calls during training and blocked 647 of them. The adapters, GGUF files, RL tasks and plugin are in a Hub collection.
Score gains and tool-call cuts, by harness
The evaluation used all 89 Terminal-Bench 2.1 tasks and the 51 hardest SWE-bench Pro tasks, with one attempt per task. With thinking off, the per-harness adapters beat the base model in every pair. Under pi on SWE-bench Pro the score rose from 40.0% to 64.0%, and under Hermes from 29.4% to 43.1%. On Terminal-Bench, Hermes went from 44.7% to 58.8%. On shared solved tasks, tool calls fell 56-72% on SWE-bench Pro and 53-70% on Terminal-Bench.

The merged adapter, using a weight of one half, gave up some of that. Its thinking-off mean pass@1 rose from 39.5% to 49.0% across the six pairs, with 53-73% fewer tool calls. The specialists stayed ahead on four of six pairs, most clearly under pi. With thinking on, the super adapter lifted the mean from 55.5% to 63.5%, and it tied the base at 74% under pi on SWE-bench Pro.
An adapter trained under pi also helped in other harnesses. Under OpenCode on Terminal-Bench it moved the score from 44.8% to 54.0%, and under Hermes from 46.3% to 56.1%, according to the post.
Gaps the post leaves open
Every number comes from Hugging Face’s own runs, and the post does not describe independent replication. The post also contains mismatches. The text gives pi on SWE-bench Pro as 39.2% rising to 64.7%, while the table shows 40.0% and 64.0%. The text puts OpenCode’s Terminal-Bench gain at 6.0 points, while the table shows 47.6% rising to 52.4%. The TL;DR’s 39% for pi on Terminal-Bench is 40.0% in the table. The 55% headline is not a row in the main table, which lists 56.5% for pi.
The authors also flag that the merge weight was picked on Terminal-Bench 2.1, so those merged numbers are “slightly optimistic”. They say the pi-versus-Hermes-versus-OpenCode comparisons are not valid, because pi drops earlier reasoning turns and the other two keep them.
Outside the training distribution the picture is thinner. On Zapier’s AutomationBench (300 tasks), the pi adapter moved pass@1 from 23.0% to 27.0%, and the merged adapter only to 23.7%. The Hermes adapter fell to 17.7%. The post says most differences are within noise.
Hugging Face’s earlier harness-adapters post, from November 6, used smaller LiquidAI models. It said its results came from smoke runs and published no base-versus-adapter accuracy table. It put the cost of evaluating nine checkpoints at about $5 of GPU time on one H200 job. It also reported that its first task set was 94% build tasks. The new post is the first to give full benchmark tables, but it does not show how well the adapters survive when a harness changes its tool names or arguments, a question a reader raised on the earlier post and the authors did not answer there.
The post also notes that in non-thinking mode the adapters write short text before tool calls, which the authors call reasoning leakage. They say it does not invalidate the results and plan to try pure thinking models and hybrid training next.
Who can try it now
The weights and tasks are public on the Hub. A single merged adapter, harness-hub/Qwen3.8-27B-super-adapter, is meant to run unchanged under all three harnesses. Training was done on 2×B200 per adapter, so reproducing it needs rented hardware. The post gives no price for a full training run.
Sources
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Daniel ReedAI Research, Safety & Ethics AnalystDaniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…



