Hot AI News

AI intelligence indexes: How one score and a cost per task rank models

An AI intelligence index blends many benchmarks into one number. Paired with cost per task, it explains why a model with pricier tokens can still be cheaper to use.

AI intelligence indexes: How one score and a cost per task rank models
Photo illustration: Aiholics

An AI intelligence index is a single score built from several benchmark tests, so that models can be ranked without reading a dozen separate leaderboards. Cost per task is its companion number: the average amount it costs a model to finish one test question. Together they let a reader compare quality and real spending in one view.

Why a cheaper token does not always mean a cheaper job

Picture two taxis. One charges $3 per mile but takes the direct route. The other charges $2 per mile but wanders. The direct taxi can still give the lower fare. AI models work the same way: the price per token (a small chunk of text) is only the rate, and the model decides how many tokens a job burns.

That gap is central to Google’s launch of Gemini 4 Neon. As we reported in our coverage of the Neon launch, its per-token price is higher than Carbon’s, yet Google says a task costs 32% less to run on Neon at the default thinking level. Those are Google’s figures, and the announcement does not publish the underlying per-task costs in text.

Advertisements

Folding many tests into one number

A touchscreen table shows six AI benchmark tasks converging into a single violet gauge.
Photo illustration: Aiholics

A single benchmark measures one skill, such as fixing software bugs or answering science questions. A composite index runs a set of benchmarks and combines the results as a weighted average. Each test gets a share of the total, so a model cannot look strong by excelling at only one thing.

The best-known example is the Artificial Analysis Intelligence Index, published by the independent firm Artificial Analysis. Its current version, 5.0.1, averages nine evaluations in four equally weighted categories of 25% each: Knowledge Work, Coding, Science and General. The coding group includes FrontierCode-AA 2.0 and Terminal-Bench 4.0. The general group includes HLE-Diamond, a hard exam-style test, and AA-Omniscience, which scores both accuracy and how rarely a model makes things up.

How the scoring is run

Artificial Analysis says every model gets the same conditions. Prompts are zero-shot, meaning instructions only, with no worked examples. Most scoring is pass@1: the model must be right on its first attempt, with no retries and no picking the best of several tries.

Reasoning models are given the largest output budget their makers disclose. Failed API calls are retried up to 30 times. Coding and agent tests run inside sandboxes, isolated virtual machines where the model can execute commands safely. The suite is mainly text and English; image, speech and multilingual ability are measured separately.

Turning the score into a price: Cost per task

In its version 4.1 update, published in June 2026, Artificial Analysis added per-task metrics. It takes the total cost, time and output tokens a model needs to run the whole index, then divides by the number of tasks. The result is the average cost, time and tokens for one task.

Cost is computed from the token counts each provider reports. Cached input tokens, text the provider has already processed and bills at a discount, are now included. Neon’s cached input rate of $0.15 per million tokens, which Google calls a 95% discount, would matter here. This is why “thinking level” settings matter: a higher setting usually produces more reasoning tokens, so it raises both the score and the cost.

Advertisements

Where the idea came from and who relies on it

Leaderboards that combine tests have been common since the first large language model comparisons. The recurring problem is saturation: once top models all score near the maximum, a test stops telling them apart. Artificial Analysis cited this when it dropped IFBench from the index in version 4.1, saying it no longer distinguished frontier models sufficiently.

The index has shifted with the field. Version 4.1 weighted agents at 34%, coding at 24%, scientific reasoning at 24% and general knowledge at 18%. Version 5.0.1 moved to the four equal groups above. Because the recipe changes, scores from different versions should not be compared directly.

Labs now quote these indexes in launch posts. Google says Neon scores 61 at its highest xhigh thinking setting, which it says ranks third overall. It also cites the Vals Index and Artificial Analysis’s Coding Agent Index. The same tracker pattern, a quality score plotted against price, is how many buyers shortlist models.

Limits to keep in mind

A composite score hides trade-offs. Two models can reach the same total with very different strengths, and the weights are a judgment call by the index’s authors. A model that suits your workload may rank lower overall.

Cost per task is an average over the index’s own tasks. Your tasks may be longer, shorter or more tool-heavy, and your cost will differ. Provider prices, caching behavior and thinking settings also change over time.

Vendor-chosen comparisons deserve extra care. A lab usually highlights the benchmarks and settings where its model looks best, and comparing one model at its top setting with a rival at its default is not like for like. Independent runs by the index’s publisher are more useful than a launch chart.

What to watch next

  • Whether independent trackers publish Neon’s cost per task at each thinking level, which would test Google’s 32% claim.
  • New index versions, since benchmark sets are replaced as they saturate.
  • Whether time per task and tokens per task become standard in launch posts, next to price per million tokens.
  • Whether buyers test models on their own workloads before trusting any single index score.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Daniel ReedAI Research, Safety & Ethics Analyst

Daniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like