SageMaker streaming gets per-use billing, so Marketplace models can charge by audio processed

Amazon Web Services has added usage-based metering to SageMaker AI bidirectional streaming, so model providers on AWS Marketplace can bill for what a live session consumes. Deepgram is the launch partner, covering speech-to-text and text-to-speech.

A second WebSocket carries the billing data

The problem, as AWS describes it, is that existing Marketplace metering for SageMaker real-time inference relies on a unit count in an HTTP response header. That works when the billable quantity is known as the response is sent. A streaming session can last 30 seconds or 30 minutes, and the total is only known when it ends, so a header cannot report usage along the way.

The fix is a companion connection. When a session starts, SageMaker AI opens a data WebSocket to the model server, as before. If the server answers the upgrade with the header X-Amzn-SageMaker-Metadata-Stream-Supported: true, SageMaker opens a second WebSocket at /invocations-bidirectional-stream-metadata by default. Both connections share an X-Amzn-SageMaker-Request-Id so the server can match metering to the session. Providers can override the path with X-Amzn-SageMaker-Metadata-Invocation-Path.

Each metering record is one JSON text frame, with the dimension inference.count and a ConsumedUnits number. AWS gives an example: a product that charges per second of audio reports 12 minutes of metered audio as ConsumedUnits: 720. An optional ClientToken of up to 64 characters makes duplicates on the same connection drop silently.

The limits are tight: 512 bytes per message, one message per second per connection, text frames only, and exactly one record per frame. AWS recommends sending records incrementally, which limits how much usage goes unreported if a session ends unexpectedly.

Metering can fail without stopping inference

A provider that does not send the opt-in header gets no metering for the software charge. A second header, X-Amzn-SageMaker-Metadata-Stream-Required, defaults to false. In that default, if the metadata connection fails or drops, inference carries on in data-only mode and metering stops. Set it to true and the data stream is terminated instead. AWS frames this as a trade between inference availability and metering continuity, and says it does not guarantee that malformed or dropped records are billed. Malformed messages are dropped either way, and the metadata connection stays open.

For customers, the software charge flows into the AWS bill, with private offers for negotiated rates. Endpoint infrastructure charges remain separate and continue while the endpoint is provisioned. Providers choose per-inference pricing when they create a listing: one per-unit price for real-time inference, shown per request with up to three decimal places. A released listing cannot switch between hourly and per-inference pricing. Customers can subscribe to a Deepgram streaming model such as Nova-3 Monolingual Speech-to-Text Streaming; the model server calculates metering with no client-side setup, according to AWS.

Why billing had to follow the stream

The technical point is that SageMaker’s streaming path was built as a pipe, not as a ledger. According to AWS’s November 2025 launch notice, a client opens an HTTP/2 connection to the SageMaker runtime and SageMaker creates the WebSocket to the container. An AWS startups post adds that this runs over port 8443 and that Deepgram’s Nova-3 speech-to-text and Aura-2 text-to-speech models run on it. A long-lived stream has no natural end-of-request moment at which to count units, which is why a separate channel was needed.

The design also leaves most of the accounting to providers. They define what counts as metered audio, and AWS says an open connection lasting 12 minutes does not by itself define the billable quantity. That puts the burden of correct counting on the model server, and AWS tells providers to test interrupted sessions, metadata loss and malformed messages against Marketplace reports before release. It points to a usage type of the form USE1-SMMP:inference.count.m.i.c for reconciliation.

Deepgram announced its SageMaker integration on November 30, 2025, so this is the next step in a year-long series: streaming first, then billing that matches it.

What the announcement leaves open

The post does not list prices for any Deepgram product, nor which Regions support the metadata stream. It also gives no reporting delay, saying only that sellers should allow time for usage reports to populate. The claim that metering aligns charges with consumption is AWS’s own, and the blog offers no customer billing data to check it. For now the practical audience is model providers weighing a Marketplace listing, and buyers of streaming speech models who want usage to appear on one AWS bill.