Amazon’s QA Studio adds team roles and a flaky-test score

Amazon Web Services has added two features to QA Studio, its reference solution for AI-driven software testing built on Amazon Nova Act: applications that report a 0-100 health score, and team-based access with five roles. The additions are described in Part 3 of the series on the AWS Machine Learning blog.

Applications that report their own health

In QA Studio, an application stands for the system under test, such as a website or a web app front end. Creating one takes a name, a base URL and, optionally, environments such as development, staging and production. The natural-language tests from Part 1 (called use cases) and the regression suites from Part 2 belong to an application and inherit its ownership.

Every execution is written to Amazon CloudWatch as a custom metric, split by application and by environment. The application page shows the pass rate, the number of executions and the current failure count. It also shows a health score from 0 to 100. According to AWS, the score starts from the pass rate and subtracts a penalty for flaky tests, so a suite that passes but flips between pass and fail does not look perfect.

The metrics tab charts executions, successes, failures and average duration over 7 days (hourly) or 30 days (daily), and can be filtered to one environment. A flaky-tests view ranks use cases by how often they flipped over the last seven days. A recent-failures view links each failed run to the Nova Act trajectory logs and session recordings from Part 1.

Five roles and a ceiling rule for sharing

Each application belongs to exactly one team, and other teams do not see it. Roles are cumulative: Viewer has read-only access, Executor can run tests, Editor can change use cases and suites, Team Lead manages members and creates applications, and Admin is a studio-wide role that covers every team. Access to a test follows the application that contains it, so permissions are managed per application rather than per test.

Teams can share an application with another team at view, execute or full level. A user’s effective permission is the lower of their own role and the grant. In AWS’s example, a view-level grant leaves even an Editor on the receiving team with read-only access.

Under the hood, each team maps to Amazon Cognito groups, one per role. An AWS Lambda authorizer turns those groups into API scopes on every request, and DynamoDB stores teams, memberships, ownership and share grants. The backend checks both the role’s scope and whether the caller’s team owns or has been shared the application. AWS notes that the OAuth client credentials flow from Part 2 fits the same model, so a CI pipeline can run as a machine identity with limited scopes.

Who can use it, and what is not established

QA Studio is a reference solution, not a managed service. Teams deploy it in their own AWS account with the AWS CDK from the QA Studio GitHub repository, and AWS says test data, results and metrics stay inside that account. After deployment an administrator creates teams, assigns roles and assigns applications. The post gives no pricing for QA Studio itself, so costs depend on the AWS services it uses and on Nova Act usage.

The post does not say how the flaky-test penalty is calculated, so the health score cannot be reproduced or compared across tools. The screenshots come from a small demo deployment with four users and two teams, and the post offers no data on how the system performs at larger scale. AWS’s descriptions of the benefits are its own and were not independently tested.

Access control for test agents arrives as agent governance grows

The angle here is competition and market context. Nova Act is aimed at browser-based UI workflows. AWS’s product page says it is powered by a custom Nova 2 Lite model and mixes natural language with Python, and it quotes customers: Hertz says QA that took weeks now takes hours, and Leo Commerce says test scenarios that took weeks now take minutes. Those are customer statements published by AWS, not independent measurements.

The release also fits a broader move toward giving agents owners and limits. IBM said on October 27 that Agent Identity and HashiCorp Vault 2.2 are generally available, as we reported, citing a Gartner projection that the average Fortune 500 company will run more than 150,000 agents by 2028, while only 13% of organizations consider their agent governance adequate. IBM’s tools address agents across a company; QA Studio addresses one narrow case, testing agents, using Cognito and IAM-style scopes on AWS.

For teams already on AWS, the practical change is that a shared testing platform no longer has to be one flat list that every squad can edit. For teams on other stacks, QA Studio is a sample to adapt, not a product to buy. AWS has also been adding agent tooling elsewhere, such as the sandboxed exploit testing in AWS Continuum.