Gemini 4 Argon tops a new Java migration test, but runtime bugs sink most agents

IBM and Kaggle have opened a public leaderboard for migrating enterprise Java applications between frameworks. In the first ranking, Gemini 4 Argon with Google’s Antigravity harness leads at a 75.2% smoke score across 204 tasks.

Ten setups ranked on 204 migration tasks

The leaderboard is built on ScarfBench, IBM’s benchmark for moving Java applications among Jakarta EE, Quarkus and Spring. It checks whether a migrated application builds, starts and behaves as before, using both focused workloads and whole applications. IBM first introduced the benchmark in March 2026, and its announcement said each workload was manually implemented and validated by experienced developers.

The main metric is the smoke percentage, which gives partial credit and weights larger test suites more heavily. The board also reports an all-pass percentage (tasks where the full test suite passes) and a mean score. According to IBM, the units ranked are combinations of a model and an agent harness, not models alone. In its words, the results “should therefore be read as model/harness results, not as measurements of the underlying model alone.”

Ten configurations were evaluated. Smoke scores, as IBM lists them:

• Gemini 4 Argon with Antigravity: 75.2%
• Claude Opus 5.5 with Claude Code: 70.5%
• Claude Opus 5 with Claude Code: 69.8%
• GPT-6.1 Sol with Codex: 68.4%
• Gemini 3.8 Flash with mini-swe: 66.0%
• GPT-6 Astra with Codex: 65.4%
• Qwen 3.8 Max with qwen-code: 63.7%
• Kimi K3 with kimi-code: 61.1%
• Gemini 3.1 Pro with OpenHands: 39.5%
• Gemma 4 26B-A4B with mini-swe: 24.5%

The order changes on the stricter metric. Claude Opus 5.5 with Claude Code has the highest all-pass rate, at 72.1%. IBM also ran a sensitivity check: four tasks earned zero behavioral credit from every configuration, and the ranking held when they were removed, leaving 200 tasks.

Code that compiles but behaves differently

IBM classified 745 failing runs. About 72% were runtime behavioral failures, about 20% were compile or packaging failures, and about 8% were boot or dependency-injection crashes. The post says behavioral fidelity separated the configurations more than compilation did.

That matches the design of the benchmark, which does not stop at a clean build. An agent that rewrites annotations and configuration until the compiler is satisfied can still change how persistence, security or injection behaves. For teams weighing automated modernization, that is the costly kind of error, because it surfaces in testing or production rather than in the build log.

Efficiency also varied. Claude Opus 5 with Claude Code used the fewest median turns, 32, while Gemini 3.8 Flash with mini-swe used the most, 79, according to IBM.

What the first ranking cannot yet tell you

The leaderboard is a view of results, not yet an open contest. IBM says public model submission is not available at launch but is on the roadmap. Kaggle, which consulted on the evaluation design, independently reproduced the benchmark on its platform and sponsored the compute, is to maintain it and add new models. IBM does not mention prizes.

Several limits follow. Because each score mixes a model with its harness, the gap between 75.2% and 70.5% cannot be attributed to the models themselves. The harnesses differ too: Antigravity, Claude Code, Codex, mini-swe, qwen-code, kimi-code and OpenHands are not interchangeable. The Gemini 3.1 Pro result (39.5%) with OpenHands and the Gemma result with mini-swe may reflect that pairing as much as the model.

The partnership post is IBM’s own write-up, and the Kaggle benchmark page itself returned no readable detail when checked, so the numbers rest on IBM’s account plus Kaggle’s stated reproduction. The March announcement did not give the number of applications, a license or earlier results, and the new post does not name a separate dataset beyond those developer-verified workloads.

Readers who want to try the benchmark on their own agents can look to the documentation and runtime CLI that IBM linked at launch, while the Kaggle leaderboard shows the current standings. IBM has also been working on agent governance, as in its push to give every AI agent an owner, an ID and a kill switch, but this benchmark covers a narrower question: whether an agent can change a Java framework without changing what the application does.