Clef added
The newest model in the System One benchmarks is Clef.
AIM System One Index
System One models, also called decision models, return a typed decision with a probability for each option in one pass instead of generating text token by token. How AIM System One Index works
Best model
autojev-27b
AIM System One Index 59
Fastest model
Laya
30 ms per decision
Cheapest model
eikos-4b
$0.014 per 1k decisions
AIM System One Index
System One Models Leaderboard
System One models ranked by the AIM System One Index.
# | Model | Index | AIM-AI-Bias (%) | AIM-Decision Browser (%) | AIM-Decision Classification (%) | AIM-Decision Browser time (s) | Cost per 1k decisions ($) | Latency per decision (ms) |
|---|---|---|---|---|---|---|---|---|
| 1 | autojev-27b | 59 | 95.3 | 68 | 48.4 | 6.9 | 0.109 | 223 |
| 2 |  Glide | 58 | 90 | 72 | 48.3 | 46.8 | 0.479 | 15,686 |
| 3 |  Solar Decide 2 of 3 benchmarks | 58 | 99.4 | - | 46 | - | 0.046 | 7,231 |
| 4 | decider-12b | 57 | 93.8 | 32 | 54.4 | 3.1 | 0.06 | 127 |
| 5 | eikos-27b | 56 | 81.3 | 74 | 46.2 | 6.8 | 0.075 | 200 |
| 6 |  shisa-de-1 2 of 3 benchmarks | 56 | 85.9 | - | 48.9 | - | 0.032 | 65 |
| 7 |  Tev1-4B 2 of 3 benchmarks | 55 | 82.8 | - | 48.5 | - | 0.338 | 682 |
| 8 | Kev-27b | 53 | 65.6 | 80 | 44.7 | 7.7 | 0.165 | 375 |
| 9 |  Jev 1.13 | 52 | 83.8 | 34 | 47.9 | 4.2 | 0.039 | 218 |
| 10 | decider-4b | 51 | 82.8 | 38 | 44.4 | 2.5 | 0.022 | 50 |
Page 1 of 3
Cost$0.109 / 1k
Latency223 ms
Context-
AIM-AI-Bias (%)
95.3
AIM-Decision Browser (%)
68
AIM-Decision Classification (%)
48.4
Cost$0.479 / 1k
Latency15,686 ms
Context-
AIM-AI-Bias (%)
90
AIM-Decision Browser (%)
72
AIM-Decision Classification (%)
48.3
Cost$0.046 / 1k
Latency7,231 ms
Context-
AIM-AI-Bias (%)
99.4
AIM-Decision Classification (%)
46
Cost$0.060 / 1k
Latency127 ms
Context-
AIM-AI-Bias (%)
93.8
AIM-Decision Browser (%)
32
AIM-Decision Classification (%)
54.4
Cost$0.075 / 1k
Latency200 ms
Context-
AIM-AI-Bias (%)
81.3
AIM-Decision Browser (%)
74
AIM-Decision Classification (%)
46.2
Cost$0.032 / 1k
Latency65 ms
Context-
AIM-AI-Bias (%)
85.9
AIM-Decision Classification (%)
48.9
Cost$0.338 / 1k
Latency682 ms
Context-
AIM-AI-Bias (%)
82.8
AIM-Decision Classification (%)
48.5
Cost$0.165 / 1k
Latency375 ms
Context-
AIM-AI-Bias (%)
65.6
AIM-Decision Browser (%)
80
AIM-Decision Classification (%)
44.7
Cost$0.039 / 1k
Latency218 ms
Context-
AIM-AI-Bias (%)
83.8
AIM-Decision Browser (%)
34
AIM-Decision Classification (%)
47.9
Cost$0.022 / 1k
Latency50 ms
Context-
AIM-AI-Bias (%)
82.8
AIM-Decision Browser (%)
38
AIM-Decision Classification (%)
44.4
Page 1 of 3
AIM System One Index vs time per decision
Higher and further left is better.
AIM System One Index vs time per decision
*AIM System One Index, as explained in How AIM System One Index works. Time: mean per AIM-Decision Classification question.
Cost per 1k decisions
AIM-Decision Classification cost of each System One model, on a hosted API or self-hosted.
Cost per 1k decisions
*Hosted API: list price times the input tokens each API reported (Solar Decide as billed). Self-hosted: A100 rental at $1.673 per hour on each producer's own server.
Individual benchmarks
AIM-Decision Browser
Tasks completed out of 50 on the same browser tasks.
Browser tasks completed out of 50
*One attempt per task.
AIM-Decision Classification
1,655 single-decision questions in four tasks: score against cost per 1,000 decisions.
Score vs cost and latency
*Score: average of the two main tasks. Cost per 1,000 decisions: API price or bill for hosted models, A100 rental time for self-hosted models. Latency: mean seconds per decision.
How AIM System One Index works
AIM System One Index combines each model's scores on AIM-Decision Browser, AIM-Decision Classification and AIM-AI-Bias.
Each score becomes a z-score, its distance from the benchmark's average in standard deviations, measured against the models that have all three scores. The index is 50 + 10 × the average of a model's z-scores, so 50 is an average model and every 10 points is one standard deviation. A missing score is skipped; the leaderboard shows how many of the three scores a model has.
Latency and cost are shown for reference and do not count toward the index.
Cost and latency per decision come from AIM-Decision Classification: A100 rental time for self-hosted models, API price for the others.
Benchmarks We Used
AIMultiple
AIM-Decision Browser50 browser tasks, one attempt each. Its success rate counts toward the index.AIMultiple
AIM-Decision Classification1,655 single-decision questions in four tasks: security advisory severity, FDA recall class, paper topic and browser task done check. Its score counts toward the index.AIMultiple
AIM-AI-BiasMultiple-choice bias questions. Its score, the share of unbiased answers, counts toward the index.Recent Updates
Models most recently added to the System One benchmarks.
Cloudflare
Clef
New model added to the System One benchmarks.
Cloudflare
Clef Flash
New model added to the System One benchmarks.
Fastino
Glide
New model added to the System One benchmarks.
Mapika
decider-12b
New model added to the System One benchmarks.