Span-02 added
The newest model in the System One benchmarks is Span-02.
AIM System One Index
v1.2System One models, also called decision models, return a typed decision with a probability for each option in one pass instead of generating text token by token. How AIM System One Index works
Top model
AutoJev-27B
AIM System One Index 60
Fastest model
Laya
30 ms per decision
Cheapest model
Decider Flash V1.2
$0.007 per 1k decisions
AIM System One Index vs time per decision
*AIM System One Index, as explained in How AIM System One Index works. Time: mean per AIM-Decision Classification question.
AIM System One Index
System One Models Leaderboard
System One models ranked by the AIM System One Index.
# | Model | Index | AIM-AI-Bias (%) | AIM-Decision Browser (%) | AIM-Decision Classification (%) | AIM-Decision Browser time (s) | Cost per 1k decisions ($) | Latency per decision (ms) | Context window (tokens) |
|---|---|---|---|---|---|---|---|---|---|
| 1 |  AutoJev-27B | 60 | 94.8 | 68 | 48.4 | 6.9 | 0.11 | 223 | 262,144 |
| 2 |  d1-ultra | 60 | 80.1 | 74 | 52.1 | 4 | 0.026 | 597 | - |
| 3 |  decider-12b | 60 | 93.8 | 32 | 54.4 | 3.1 | 0.06 | 127 | 262,144 |
| 4 |  GLiDE | 59 | 90 | 72 | 48.3 | 46.8 | 0.479 | 15,686 | 40,000 |
| 5 |  Solar Decide Flash 2 of 3 benchmarks | 59 | 97.6 | - | 46.2 | - | 0.046 | 1,132 | 524,288 |
| 6 | Solar Decide 2 of 3 benchmarks | 58 | 97.2 | - | 46 | - | 0.046 | 7,231 | 524,288 |
| 7 |  Jev 1.14 | 57 | 87.5 | 62 | 48.1 | 3.2 | 0.039 | 634 | - |
| 8 | GLiDE v2 | 57 | 78.1 | 72 | 49.1 | 6 | 0.096 | 473 | - |
| 9 |  Shisa DE-1 2 of 3 benchmarks | 57 | 83.9 | - | 48.9 | - | 0.021 | 65 | 262,144 |
| 10 | Shisa DE-2 | 56 | 88 | 44 | 49.8 | 17.2 | 0.255 | 1,138 | 262,144 |
Page 1 of 6
Cost$0.110 / 1k
Latency223 ms
Context262k
AIM-AI-Bias (%)
94.8
AIM-Decision Browser (%)
68
AIM-Decision Classification (%)
48.4
Cost$0.026 / 1k
Latency597 ms
Context-
AIM-AI-Bias (%)
80.1
AIM-Decision Browser (%)
74
AIM-Decision Classification (%)
52.1
Cost$0.060 / 1k
Latency127 ms
Context262k
AIM-AI-Bias (%)
93.8
AIM-Decision Browser (%)
32
AIM-Decision Classification (%)
54.4
Cost$0.479 / 1k
Latency15,686 ms
Context40k
AIM-AI-Bias (%)
90
AIM-Decision Browser (%)
72
AIM-Decision Classification (%)
48.3
Cost$0.046 / 1k
Latency1,132 ms
Context524k
AIM-AI-Bias (%)
97.6
AIM-Decision Classification (%)
46.2
Cost$0.046 / 1k
Latency7,231 ms
Context524k
AIM-AI-Bias (%)
97.2
AIM-Decision Classification (%)
46
Cost$0.039 / 1k
Latency634 ms
Context-
AIM-AI-Bias (%)
87.5
AIM-Decision Browser (%)
62
AIM-Decision Classification (%)
48.1
Cost$0.096 / 1k
Latency473 ms
Context-
AIM-AI-Bias (%)
78.1
AIM-Decision Browser (%)
72
AIM-Decision Classification (%)
49.1
Cost$0.021 / 1k
Latency65 ms
Context262k
AIM-AI-Bias (%)
83.9
AIM-Decision Classification (%)
48.9
Cost$0.255 / 1k
Latency1,138 ms
Context262k
AIM-AI-Bias (%)
88
AIM-Decision Browser (%)
44
AIM-Decision Classification (%)
49.8
Page 1 of 6
Cost per 1k decisions
AIM-Decision Classification cost of each System One model, on a hosted API or self-hosted.
Cost per 1k decisions
*Hosted API: list price times the input tokens each API reported; OpenRouter models as billed, Mercury Decide at the price of its paid route. Self-hosted: A100 rental at $1.673 per hour with the GPU fully loaded, using each producer's code. The first Ollama versions of Nimble 9B and the Tev1 models keep an earlier measurement at up to 16 parallel requests.
Individual benchmarks
Tasks completed out of 50 on the same browser tasks.
Browser tasks completed out of 50
*One attempt per task.
1,655 single-decision questions in four tasks: score against cost per 1,000 decisions.
Score vs cost and latency
*Score: average of the two main tasks. Cost per 1,000 decisions: API price or bill for hosted models, A100 rental time with the GPU fully loaded for self-hosted models. Latency: mean seconds per decision.
How AIM System One Index works
AIM System One Index combines each model's scores on AIM-Decision Browser, AIM-Decision Classification and AIM-AI-Bias.
Each score becomes a z-score, its distance from the benchmark's average in standard deviations, measured against the models that have all three scores. The index is 50 + 10 × the average of a model's z-scores, so 50 is an average model and every 10 points is one standard deviation. A missing score is skipped; the leaderboard shows how many of the three scores a model has.
Latency and cost are shown for reference and do not count toward the index.
Cost and latency per decision come from AIM-Decision Classification. Cost is the API price for hosted models and the A100 rental time with the GPU fully loaded for self-hosted models.
Benchmarks We Used
AIMultiple
AIM-Decision Browser50 browser tasks, one attempt each. Its success rate counts toward the index.AIMultiple
AIM-Decision Classification1,655 single-decision questions in four tasks: security advisory severity, FDA recall class, paper topic and browser task done check. Its score counts toward the index.AIMultiple
AIM-AI-BiasMultiple-choice bias questions, each asked with the “cannot be determined” option in every position. Its score, the share of unbiased answers averaged over those positions, counts toward the index. The first versions of GLiDE, Nimble 9B and the two Tev1 models were tested with that option last only.Changelog
16 updates- v1.2
5 models entered the index: d3-flash, d3-mini, Decider Flash V1.2, Decider V1.2, Span-02.
- v1.2
2 models entered the index: d1-ultra, Jev 1.14.
- v1.2
7 models entered the index: APUS Reflex 35B-A3B, Decision-1.0-Lux-9B, MATILDA JEV v1.5, NeoHorse-Jev-4B, Nimble 9B v2 and 2 more.
- v1.2
12 models entered the index: d1-3B, d3, Decision-2.0-Vega-27B, Decision 4B v1.6, GEV-26B-Decide and 7 more.
- v1.2
2 models entered the index: Drex v2.0, SubQ Snap.
- v1.2
3 models entered the index: Clef-Omni, Drex v1.5, Microsoft-Decision-1.
- v1.2
5 models entered the index: d1, GPT-6 Luna Decisions, Mercury Decide, pplx-decider-v1.1-27b, Solar Decide Flash.
- v1.2
AIM System One Index: AIM-Decision Classification results now count toward the index. Version updated to v1.2.
- v1.2
AIM System One Index: The index is now the mean z-score across its benchmarks on their latest data, shown as 50 + 10 × z.
- v1.1
AIM System One Index: AI bias results now count toward the index. Version updated to v1.1.