Hot AI News

Copilot can now test itself: Evals reruns your skills and MCP tools

GitHub's new Evals preview turns a Copilot setup into something testable: run a task, check the result and the steps taken, then compare before and after a change.

Copilot can now test itself: Evals reruns your skills and MCP tools
Image: Microsoft

GitHub, part of Microsoft, has put GitHub Copilot Evals into public preview. It lets developers write repeatable tests for the skills, custom agents and MCP tools in their Copilot setup, then rerun them after every change.

GitHub frames the problem simply: as prompts, models, skills and tools change, a user may no longer be getting the most out of Copilot. Evals is meant to answer that with evidence rather than a hunch.

GitHub changelog header for the Copilot app Microsoft Ignite releases
Image: Microsoft

How an eval works inside Copilot

GitHub defines an eval as “a repeatable test for AI behavior.” The developer gives an agent a realistic task and describes what a good result looks like. The eval then checks two things: the final output and how the agent got there.

The reason for repeating the test, according to GitHub, is that AI output varies from run to run, so one good answer says little about whether a customization works. After editing a skill, agent or MCP setup, the same eval can be run again to catch regressions.

Copilot can now test itself: Evals reruns your skills and MCP tools
Image: Microsoft

Users can ask Copilot in plain language or type the /eval command. GitHub lists four things it covers:

  • creating an eval for a skill, custom agent or MCP tool in a project;
  • running it and seeing which checks passed and failed, with each run needing the user’s approval before it starts;
  • finding out why a check failed so the user can decide what to change;
  • comparing runs from before and after a customization change.

Under the hood, the feature uses Vally, which GitHub describes only as an extensible local eval tool, as its evaluation engine. The announcement gives no further detail on how Vally works or what kinds of checks it supports.

Copilot can now test itself: Evals reruns your skills and MCP tools
Image: Microsoft
Advertisements

Three places to install it, three paid plans

The preview is open to Copilot Individual, Copilot Business and Copilot Enterprise. GitHub does not mention a free tier, extra pricing or regional limits, so those questions are unanswered in the post.

There are three install routes. In Visual Studio Code, it is an extension from the Visual Studio Marketplace. In Copilot CLI, it is a plugin installed by following GitHub’s instructions. In the GitHub Copilot app, users open Customize, add the github/evals marketplace and install the plugin. Feedback goes through an issue or the GitHub Community discussion.

JetBrains is not on the list, even though GitHub shipped a separate set of Copilot updates for that IDE on the same day.

Why testing the setup matters more now

The angle here is what to watch, and the context is that Copilot is increasingly something users assemble rather than simply install. Evals appeared on the same day as a batch of Ignite releases that widen what a Copilot agent can touch. According to GitHub’s Copilot app release notes, the app now offers a public preview of Azure DevOps integration, where agents can create branches, open pull requests and merge once checks pass, and a browser-use preview that is off by default. GitHub Enterprise Server support reached general availability and requires app version 1.2.0 or later. More reach for agents means more setups worth checking.

Administrators are also shaping what users can load. Our earlier report on a new policy that lets Copilot admins bar MCP servers and skills that do not come from plugins shows that plugins are becoming the managed route for customization. Evals itself ships as a plugin in the CLI and the app, so it sits on that same path.

What would confirm the value of the preview? Three things are worth watching. First, whether GitHub documents what Vally’s checks can actually verify about an agent’s steps, since that is the hard part of grading agents. Second, whether evals prove stable enough, given GitHub’s own point that outputs vary between runs. Third, whether the feature extends to JetBrains and to team-level use rather than the individual focus of the announcement, which opens with the aim of optimizing Copilot “for yourself.”

Advertisements

What the announcement leaves unsaid

The post is a changelog entry, not a technical paper. It offers no benchmark, no example eval results, no limits on how many runs a plan allows and no statement of what a run costs in premium requests or tokens. Because each run is an agent doing real work, that could matter for teams that want to run evals often, but the source does not establish it either way. GitHub’s claims about what Evals helps users do are its own and have not been independently verified.

For developers who already maintain custom skills or MCP servers, the practical change is modest but real. Before, checking whether an edit helped meant trying it by hand and trusting memory. Now there is a saved task, a pass or fail per check, and a way to compare two runs. Whether that holds up in daily use is something the public preview will have to show.

Copilot’s other news from the same day, including the app’s Azure DevOps and browser features, points the same way: more autonomy for agents, and a growing need to measure what they do.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Leo MartinsAI Tools, Prompts & Practical AI Expert

Leo Martins is the AI Tools & Practical AI Expert at Aiholics, focused on helping readers use artificial intelligence to improve productivity, creativity and everyday work.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like