Google’s new engineering exam stumps every frontier AI model: Best score is 48.3%

Google’s AI & Infrastructure team has introduced Physical-Engineering’s Last Exam (PLE), a benchmark built with Mercor to test AI agents on mechanical, electrical, civil and structural engineering work. According to Google’s announcement, no frontier model scored above 50%.

Eighty-five projects, each with dozens of files

PLE contains 85 long-horizon tasks: 30 in mechanical engineering, 27 in electrical, 18 in civil and 10 in structural. Instead of answering a prompt, an agent works inside a containerized project folder built on Mercor’s Archipelago environment. Each folder mimics what an engineer receives on the job, with an average of 61 files per task and 5,170 across the benchmark. The files include PDFs, spreadsheets, emails, 2D CAD drawings, 3D BIM models and document revisions.

The agent has to find the governing building codes and physical limits, run calculations and produce deliverables that a licensed reviewer could approve. Google says 44 tasks were written entirely by practicing engineers, including the project folders, prompts and grading rubrics. For the other 41, engineers designed the initial task and an agent pipeline assembled the environment and grader from licensable engineering standards. Engineers then validated every one of those tasks and made them harder.

Scoring is gated. Agents earn points only if their design first passes every hard safety check, and the grading also weighs code compliance and cost efficiency.

Constraints mostly met, safety gates mostly failed

Google tested 14 frontier models across seven public agent harnesses: Antigravity, Claude Code, Codex, Hermes, OpenCode, OpenHands and Pi. On the private test set, the best model on its native harness reached a mean score of 48.3%.

That is the gap the benchmark is designed to expose. An agent can get most details right and still hand over a design that a reviewer would reject. Google also reports that the choice of model matters far more than the harness: scores varied by up to 50 points between models, but the median spread across harnesses for the same model was 3 points. Harnesses did change run costs significantly, according to the company.

How to enter, and what Google has not said

Google is releasing 29 public tasks, the grading scripts and the evaluation container on GitHub, and a paper is on arXiv. Developers can submit their own agent setup for scoring on the private test set at ple-bench.com. Thirteen Google-sponsored models run free on the organizers’ servers. Google says prize details are coming soon but gives no amounts or dates.

The results are Google’s own and have not been independently verified. The announcement does not name the 14 models or give a per-model ranking, so readers cannot yet see how Gemini compares with rivals. The post also does not say how the private set differs in difficulty from the 29 public tasks. The public tasks are only about a third of the benchmark, which limits what outside teams can inspect.

A benchmark family that now reaches the physical world

Google positions PLE next to two earlier exams: Humanity’s Last Exam for general knowledge and Agents’ Last Exam for digital professional work. Humanity’s Last Exam, released in January 2025 by the Center for AI Safety and Scale AI, consists of 2,500 questions written by subject experts, per its paper. That is a bank of short questions, whereas PLE has 85 tasks that each require an agent to work through dozens of files.

The difference matters for how scores are read. Google notes that frontier models exceed 90% on many academic benchmarks, mostly because those test textbook-style questions or small, self-contained problems. A sub-50% ceiling on PLE suggests the shortfall lies in sustained, multi-file work under hard constraints, not in raw knowledge. The 41.3% pass rate on safety checks also points to a practical limit: in a field where licensed engineers sign off on designs, a deliverable that fails one gate is not a partial success.

Google frames the goal as supporting engineers, with a licensed professional keeping review and control, as infrastructure demand outpaces financing. The benchmark does not suggest that agents can replace that review. Its first results suggest the opposite: the safest reading of 48.3% is that human sign-off remains essential.