Microsoft tests Copilot on fake companies, saying 39% better data quality and no customer data

Microsoft researchers have described a platform that tests Copilot on fully synthetic companies instead of real customer data. In its research blog post, Microsoft says the synthetic data scores 39% higher on its composite quality metric than the team’s first approach.

Three invented companies with hundreds of employees

The Industry Evaluation Data Generation Platform produces simulated Microsoft 365 workspaces, which Microsoft calls tenants. Each has users, meetings, emails, documents and an org chart. Microsoft says each environment holds hundreds of users with full communication histories, and that the environments change over time. Email threads run for weeks and reorganizations happen. That matters for time-aware prompts such as asking what a team decided last Thursday.

So far there are three environments:

• a pharmaceutical company, with regulatory submissions, clinical trial discussions and safety review meetings;
• a capital markets firm, with deal pipelines, compliance messages and client portfolio reviews;
• an IT and software company, with sprint planning, code reviews and launch coordination.

To make them realistic, Microsoft drew on three public sources. O*NET, the US Department of Labor’s occupational database, shaped roles and the mix of work activities. Online job postings supplied titles and reporting lines. SEC filings informed company size, business units and financial context. Microsoft says its legal and policy organization reviewed and approved all of them.

A six-stage pipeline writes the test questions

An environment alone does not test anything, so the team also built a pipeline to write the queries. According to Microsoft, it starts from a large library of generic workplace tasks and filters it for a given industry. It adapts the wording to domain language, then uses AI to generate queries grounded in the synthetic data. Those are rewritten to sound more human and checked for accuracy and uniqueness.

The environments are scored on four dimensions: structural fidelity, contextual relevance, behavioral realism and content safety. Microsoft says some industry environments scored above 0.9 out of 1.0 on the composite metric. It also says the whole process, once a multi-day manual job, now runs automatically and gives identical results from the same configuration. The company estimates savings of about $100,000 per evaluation dataset and $300,000 for the project overall.

Privacy is the point, but the proof is internal

The privacy case is the core of the pitch. Microsoft states that “no real customer data is used at any stage of the pipeline,” and says generated content is screened for harmful or personally identifiable material and for impersonation of real people. Its safeguards include privacy-preserving aggregation where real statistics are referenced, legal review of sources and provenance tracking. Because of this, Microsoft says, teams can reuse the datasets without privacy review overhead.

The public inputs are broad by design. According to the O*NET OnLine site, the database is sponsored by the US Department of Labor and lists 1,016 occupation titles and codes. That gives the generator a wide, documented range of roles to draw from.

Synthetic data is a crowded field. Our earlier analysis of synthetic data cited a market of about $1.3 billion in 2024 and more than 190 startups focused on it. Microsoft’s approach differs in one respect: it builds the data for its own testing instead of selling it.

The post leaves several things open. The 39% gain and the 0.9 scores come from Microsoft’s own internal framework. The post does not say how scores relate to Copilot’s accuracy in practice, and it publishes no Copilot results. It also does not say whether the datasets will be shared outside Microsoft. The post says they sit in an internal shared catalog, so customers cannot use them yet. Nor does it explain how the team checks that fabricated organizations avoid resembling real ones. The post presents no independent check of the cost savings either.

Agentic testing is the next hurdle

Microsoft says the work now moves toward multi-turn agentic interactions, environments that evolve over months or years, and evaluation of AI that takes actions rather than only writing text. For readers, the practical change is indirect. Copilot changes should now be measured against a repeatable baseline, which should make regressions between releases easier to spot. How much that helps real users will depend on results Microsoft has not yet published. The same standard already covers some regulated settings, since the pharma and finance environments mirror industries where mistakes are costly. The company’s wider Copilot push is covered in our reports on its latest quarterly results.