Copilot can now test itself: Evals reruns your skills and MCP tools
GitHub's new Evals preview turns a Copilot setup into something testable: run a task, check the result and the steps taken, then compare before and after a change.

GitHub, part of Microsoft, has put GitHub Copilot Evals into public preview. It lets developers write repeatable tests for the skills, custom agents and MCP tools in their Copilot setup, then rerun them after every change.
GitHub frames the problem simply: as prompts, models, skills and tools change, a user may no longer be getting the most out of Copilot. Evals is meant to answer that with evidence rather than a hunch.

How an eval works inside Copilot
GitHub defines an eval as “a repeatable test for AI behavior.” The developer gives an agent a realistic task and describes what a good result looks like. The eval then checks two things: the final output and how the agent got there.
The reason for repeating the test, according to GitHub, is that AI output varies from run to run, so one good answer says little about whether a customization works. After editing a skill, agent or MCP setup, the same eval can be run again to catch regressions.

Users can ask Copilot in plain language or type the /eval command. GitHub lists four things it covers:
- creating an eval for a skill, custom agent or MCP tool in a project;
- running it and seeing which checks passed and failed, with each run needing the user’s approval before it starts;
- finding out why a check failed so the user can decide what to change;
- comparing runs from before and after a customization change.
Under the hood, the feature uses Vally, which GitHub describes only as an extensible local eval tool, as its evaluation engine. The announcement gives no further detail on how Vally works or what kinds of checks it supports.

Three places to install it, three paid plans
The preview is open to Copilot Individual, Copilot Business and Copilot Enterprise. GitHub does not mention a free tier, extra pricing or regional limits, so those questions are unanswered in the post.
There are three install routes. In Visual Studio Code, it is an extension from the Visual Studio Marketplace. In Copilot CLI, it is a plugin installed by following GitHub’s instructions. In the GitHub Copilot app, users open Customize, add the github/evals marketplace and install the plugin. Feedback goes through an issue or the GitHub Community discussion.
JetBrains is not on the list, even though GitHub shipped a separate set of Copilot updates for that IDE on the same day.
Why testing the setup matters more now
The angle here is what to watch, and the context is that Copilot is increasingly something users assemble rather than simply install. Evals appeared on the same day as a batch of Ignite releases that widen what a Copilot agent can touch. According to GitHub’s Copilot app release notes, the app now offers a public preview of Azure DevOps integration, where agents can create branches, open pull requests and merge once checks pass, and a browser-use preview that is off by default. GitHub Enterprise Server support reached general availability and requires app version 1.2.0 or later. More reach for agents means more setups worth checking.
Administrators are also shaping what users can load. Our earlier report on a new policy that lets Copilot admins bar MCP servers and skills that do not come from plugins shows that plugins are becoming the managed route for customization. Evals itself ships as a plugin in the CLI and the app, so it sits on that same path.
What would confirm the value of the preview? Three things are worth watching. First, whether GitHub documents what Vally’s checks can actually verify about an agent’s steps, since that is the hard part of grading agents. Second, whether evals prove stable enough, given GitHub’s own point that outputs vary between runs. Third, whether the feature extends to JetBrains and to team-level use rather than the individual focus of the announcement, which opens with the aim of optimizing Copilot “for yourself.”
What the announcement leaves unsaid
The post is a changelog entry, not a technical paper. It offers no benchmark, no example eval results, no limits on how many runs a plan allows and no statement of what a run costs in premium requests or tokens. Because each run is an agent doing real work, that could matter for teams that want to run evals often, but the source does not establish it either way. GitHub’s claims about what Evals helps users do are its own and have not been independently verified.
For developers who already maintain custom skills or MCP servers, the practical change is modest but real. Before, checking whether an edit helped meant trying it by hand and trusting memory. Now there is a saved task, a pass or fail per check, and a way to compare two runs. Whether that holds up in daily use is something the public preview will have to show.
Copilot’s other news from the same day, including the app’s Azure DevOps and browser features, points the same way: more autonomy for agents, and a growing need to measure what they do.
Sources
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Leo MartinsAI Tools, Prompts & Practical AI ExpertLeo Martins is the AI Tools & Practical AI Expert at Aiholics, focused on helping readers use artificial intelligence to improve productivity, creativity and everyday work.
You may also like

Copilot app adds multi-repo Projects and a computer use mode that clicks through desktop apps

Copilot CLI can now sign in with Entra, and bill Microsoft 365 instead of GitHub

Copilot sessions in JetBrains can now run in a cloud sandbox or follow you to VS Code
