Decision models, also called System One models,1 score a fixed set of options instead of generating text token by token. We tested 22 decision models on 1,655 classification questions, such as rating a security flaw or a product recall, and ran 19 of them on 50 browser tasks.
Classification benchmark results
- decider-12b scored 46.4 on security flaws and 62.4 on recalls, at $0.06 per 1,000 decisions. No decision model scored significantly higher on either task.
- Jev 1.13 cost $0.039 per 1,000 decisions and scored 47.9, about a point below Kev-9B and Nimble 9B, which cost about 7 and 9 times more.
- Eikos-27B, with 27 billion parameters, cost $0.075 per 1,000 decisions. Tev1 4B, with 4 billion, cost $0.34.
- GLiDE scored 46.5 on security flaws against decider-12b’s 46.4, but took 15.7 seconds per decision against 0.13 and cost $0.48 per 1,000 decisions. Clef averaged 51.0 at $0.17 through Cloudflare’s API.
Browser automation success rates
One attempt per task; the LLMs ran at their API’s low reasoning-effort setting.
- Solar Decide, Nimble 9B and the two Tev1 models are not ranked: their services accept at most 26 options per question, and 17 of the 50 tasks open on a page with more. Laya is not ranked either, because its input limits rejected requests that the other models received in full.
- Kev-27B completed 40 tasks, two fewer than Gemini 3.8 Flash and seven fewer than GPT-6 Astra, at an estimated $0.006 per completed task against $0.026 for Gemini and $0.226 for Astra.
- The decision models together completed 44 of the 50 tasks, including two of the three that Astra failed.
- decider-12b, first on classification, completed 16 browser tasks; Kev-27B, which scored below Jev on classification, completed 40.
See 10 example tasks and every model’s result on Hugging Face, and how we ran the benchmark under methodology.
Decision models vs LLMs on browser tasks
Most LLMs write their answer one token at a time, and every token costs compute and time. For most browser steps the agent only picks an operation and a control on the page.
A decision model reads the page and the options, scores each option, and returns the most likely one. It returns a probability for each option instead of generated text, so a step needs no long output. If decision models become accurate enough, the same automation could run much faster and at a fraction of LLM cost.
The decision models, as we served them, return option scores and no text, so a separate text model, Mercury 2.5,2 writes the values typed into form fields. We call it the text helper, and it was available to every model.
We ran every model on browser-use’s open-source jev-ultrafast runtime,3 which was built for Jev 1.13 from TypeSafe.4 GPT-6 Astra serves as a high-end LLM reference and Gemini 3.8 Flash as a low-cost one. All models are listed under methodology.
Five models on the same task
On Wikipedia, every model except Laya typed into the search box, the text helper wrote “Ada Lovelace”, and the model picked the right suggestion. Jev reached the article in 8.1 seconds, Gemini in 11.0, Astra in 18.9 and Kev-9B in 43.8. Laya declared the task done on the main page without taking an action.
The filter task needs two dropdowns and a search button. Jev, Gemini and Astra took the same four steps and finished in 4.6, 10.7 and 20.3 seconds. Laya clicked the site logo and declared the task done. Kev-9B failed this task in the benchmark, and in the demo it was still clicking after 11 minutes, so it is left out of this clip.
Captions show each model’s chosen operation and target, and for decision models the probability of that operation. The clips come from a separate demo run at 1× speed, not from the scored benchmark, and each final page was checked with the benchmark’s grader.
How models failed
Jev and Kev-9B together completed 26 distinct tasks. Both completed the two Wikipedia tasks and the Web Scraper dropdown task, as did both LLMs. On Books to Scrape, where each task moves from a category list to a product page, Jev completed none of the six and Kev-9B completed three. Kev-27B, Eikos-27B and AutoJev-27B completed all six.
- Jev declared the task blocked in 24 of its 33 failures, 16 times before taking any action.
- Kev-9B took a median of 9.5 actions on failed tasks, against 3 on tasks it completed. Ten attempts hit the action or decision-call limit, and five more were stopped after three actions in a row left the page unchanged.
- Kev-4B, from Kev-9B’s developer, declared the task blocked at its first decision on 29 of the 50 tasks.
- Kev-27B, Eikos-27B and GLiDE all failed seven tasks in common. They declared four of them blocked, two ScrapeMe product lookups and two Web Scraper tasks, and none of the three finished the checkbox or the calculator form.
- GPT-6 Astra failed 3 tasks. It marked a ScrapeMe category task as blocked, kept typing into the same calculator field until it was stopped, and on form task T02 the text helper returned no valid value after Astra had filled and submitted the form.
- Gemini 3.8 Flash failed 8, including the same T02 ending. Four failures reached the correct page but did not finish, two were stopped by our cost safeguard when OpenRouter did not report a fee, and one hit the action limit.
Why some models scored 0/50
GLiNER2.5-Decide
GLiNER2.5-Decide from Fastino5 declared all 50 tasks done too early, 49 of them at its first decision. On the 37 tasks that share a start page with another task, its operation probabilities differed by no more than 6 points between goals.
Fastino describes GLiNER2.5-Decide as a classifier for decisions such as ticket routing, intent, urgency and whether an agent has finished, and states that the model does not reason. The model is built for inputs of up to 512 tokens but accepts longer ones. Our first-step requests had a median of 1,915 tokens.
CLM-v0.1-8B
CLM-v0.1-8B from Contrastive-LM6 chose wait on 2,199 of its 2,288 operation decisions, so 36 attempts ran out of actions. On the same start pages, its probabilities differed by at most 13 points between goals.
Contrastive-LM states that CLM-v0.1-8B, without fine-tuning, performs on par with Jev on computer-use, gaming and tool-calling tasks, and that it accepts Jev’s request format unchanged.7 We sent it requests in that format.
Speed and success
Kev-27B, Eikos-27B and AutoJev-27B completed 68% to 80% of tasks at a median of about 7 seconds per attempt, faster than Gemini at 12.7 seconds and Astra at 17.4. decider-4b, decider-12b and Jev took 2.5 to 4.2 seconds per attempt and declared 19 to 24 tasks blocked.
GLiDE completed 72% at a median of 46.8 seconds per attempt, the slowest of any model, and 3 of its attempts hit the 900-second limit.
GLiNER2.5-Decide (0.2 seconds) declared 49 of 50 tasks done at its first decision, and Kev-4B declared 29 tasks blocked at its first.
Kev-9B took a median of 38.3 seconds per attempt, five times Kev-27B. Each Kev-9B decision took a median of 3.7 seconds of server time, against about 1 second for a full Jev API call. Request overhead added about half a second.
Kev-9B averaged 26 decisions per attempt, against about 4 for Jev, and 90% of them came on failed attempts.
Cost
Browser automation cost
† Estimate: A100 cost per input token from the classification throughput test, times the browser input tokens, plus the text helper’s fees. Method under methodology.
‡ The higher figure adds 17 requests that failed without reporting tokens, counted at 120,000 input tokens each; Fastino may not bill them.
Astra, Gemini and Jev each used 3,150 to 3,470 input tokens per decision. Astra’s input price is about 240 times Jev’s.89
At low reasoning, Astra produced 6,599 output tokens across 307 decisions, against 968,376 input tokens. Jev’s output price is $0.
At Jev’s input price, Astra’s 968,376 decision input tokens would have cost about $0.04 instead of $9.68. Kev-27B came within seven tasks of Astra for an estimated $0.25 across the 50 tasks, against Astra’s $10.63.
On these estimates, every decision model in the table that completed a task, except Kev-9B, cost less per completed task than Gemini. None cost less than Jev.
- Kev-9B cost about 3 times Gemini and 46 times Jev per completed task. Its estimated A100 cost was about $0.47 per million input tokens, 11 times Jev’s API price, and it made 1,321 decisions to Jev’s 185.
- Kev-4B’s estimated GPU cost per token was about 6.5 times its OpenRouter price.
The A100 time we rented for the Kev-9B run, 3.32 hours including setup and idle time, came to about $5.31. One Jev fee and two Gemini fees were not recorded and are missing from the table.
Classification cost
Hosted models are priced at their API list prices, except Solar Decide at its billed fees. Local models are priced per 1,000 decisions at the highest throughput we measured on one rented A100.
A rented A100 costs about $40 a day, busy or idle. At the rate we recorded, $40 buys about a million Jev decisions.
Add as preferred sourceTraining decision models
We fine-tuned Kev-9B and Laya on decisions that GLM-5.3-Flash, the labeling model, made on browser tasks from eight websites outside the benchmark. Then we ran the 50 tasks again. Each fine-tuned model is compared with its stock version, run on the same day and hardware.
The fine-tuned Kev and both Laya versions are on Hugging Face.
p comes from a paired test on the same 50 tasks (exact McNemar), and p below 0.05 counts as significant. Stock Kev completed 21 tasks in this comparison, one more than in the results above. One of Kev’s 12 gains, C02, was a browser timeout in the stock run rather than a model failure. Without it, Kev gained 11 tasks and lost 1 (p = 0.006).
- Fine-tuned Kev gained tasks on six of the nine websites, three each on Quotes to Scrape and Scrape This Site. Its results on Web Scraper, Wikipedia and the TestPages forms did not change.
- Fine-tuned Kev ended 5 attempts blocked against 12 and hit the action limit 6 times against 11. It declared an unfinished task done 7 times, against 4.
- On 41 tasks from the training websites that were kept out of training, Kev completed 21 against 13 before. On 101 single decisions from 19 of those tasks, it chose the labeling model’s operation 94% of the time, against 62% before.
- Laya’s first fine-tune chose the labeling model’s operation on 75% of the 88 decisions it could read, against 19% for stock Laya. It called a task done on a page that still needed an action 2 times instead of 53.
- In the benchmark, Laya’s first fine-tune declared an unfinished task done 36 times, once before taking any action. Stock Laya did so 32 times, 17 of them before any action.
Decision model use cases
- Checking an agent’s done signal. On the last page a browser agent saw before it declared a task done, 13 of the 22 decision models scored above 90 on telling done from not done, and AutoJev-27B, Clef and Kev-27B scored 99.5. We did not test checks on pages earlier in a run.
- Triage with fixed labels, such as rating security advisories or recalls. decider-12b beat random answers by 21 points on severity and 29 on recalls. On paper topics, a keyword classifier trained on 28,595 labelled papers scored 70.4, and no decision model scored significantly higher; the same kind of classifier scored near random on severity and recalls.
- Choosing each action in browser automation. The best decision model, Kev-27B, completed 40 of 50 tasks, against 42 and 47 for the two LLMs. Fine-tuning Kev-9B on 606 decisions from other websites raised it from 21 to 32.
On the two main classification tasks, local decision models cost an estimated $0.014 to $0.36 per 1,000 decisions on one rented GPU and averaged under a second per decision. On the browser tasks, Jev’s input price was about 1/240 of GPT-6 Astra’s.
Each answer also comes with a probability per option, which an application could use to pass uncertain cases to an LLM. We did not test how well those probabilities match actual accuracy. Seventeen of the 22 models ran on a single rented GPU; Jev, Solar Decide, GLiDE, Clef and Clef-Flash are hosted APIs.
Methodology
Browser automation benchmark
All models used the same jev-ultrafast browser runtime. At each step, it turned the current page into page text plus a numbered list of visible controls. The model chose an operation, such as click, type, select, done, or blocked, and a target from that list. The runtime carried out the action and re-read the page.
The decision models rank supplied options and return no text. When any model chose to type, the text helper wrote the value into the form field. Each attempt started in a fresh browser session.
GPT-6 Astra, Gemini 3.8 Flash, Jev, Kev-9B and Laya ran on 22 September 2026, with the browser on one Mac and Kev-9B on a rented A100 GPU.
GLiNER2.5-Decide and CLM-v0.1-8B ran on 25 September on rented RTX 4090 GPUs with Chrome on Linux. Kev-4B ran on 29 September on a rented RTX A6000 GPU. Solar Decide ran on 30 September through OpenRouter from the same Mac, and Nimble 9B and both Tev1 models ran the same day through Ollama 0.35.0 on a rented RTX A6000 GPU with Chrome on Linux.
GLiDE, Clef and Clef-Flash ran on 2 October through their providers’ APIs from the same Mac, with Chrome 154 instead of 153. Kev-27B, AutoJev-27B, Eikos-27B, Eikos-4B, decider-4b and decider-12b ran on 2 and 3 October on rented A100 GPUs with Chrome on Linux.
No request was shortened to fit a model. Solar Decide’s API and Ollama accept 2 to 26 options per question. A request outside those limits ended the attempt, as oversized requests did for Laya. Ollama accepts option descriptions only as text, so structured options were sent as compact JSON text. Tev1 ships with a 2,050-token context; we raised it to 8,194, Nimble’s own setting, so pages were not cut.
Cloudflare’s service reads only the first 2,048 tokens of the page text for Clef and Clef-Flash. We did not send longer pages, which ended one attempt for each model.
Clef, Clef-Flash, decider and Eikos require at least two options per question. A question with one option, such as a page’s only text field, was sent with that option twice, and the answer was mapped back to it.
We ran Kev-9B with the producer’s reference code in full 32-bit precision. Two optional kernel packages that speed up its base model were not installed, so the model fell back to slower reference kernels. We did not test a faster GPU setup.
CLM’s own code turns each request into text. GLiNER received the same text, with each question’s goal and rules as its prompt. Kev-4B is also served on OpenRouter by SiliconFlow at 8-bit precision.10 Run through it from the Mac on 29 September, Kev-4B completed 6 tasks.
Browser models
Low reasoning is a provider-defined effort setting. Astra’s 307 decisions included 791 reported reasoning tokens. Kev-9B11 and Kev-4B12 come from the same developer, and Nimble 9B is from Bespoke Labs.13 GLiDE comes from Fastino, the developer of GLiNER2.5-Decide,14 and Clef and Clef-Flash from Cloudflare.15
Dataset
The 50 tasks cover nine websites: eight practice sites built for scraping and testing, plus Wikipedia. The tasks include catalog navigation, search, sorting, pagination, product options, forms and article lookups. Each task starts from a fixed URL and states the end state in one sentence, for example “Switch to Science Fiction, find Dune (Dune #1), open its product details, and stop.”
Before the models ran, a scripted run completed all 50 tasks using only the runtime’s supported actions. Every task is solvable without passwords, CAPTCHAs or the Enter key.
Example tasks
Ten of the 50 tasks are published on Hugging Face with the instructions, start URL, pass conditions and every model’s result. The other 40 are withheld for later runs.
Evaluation
A task passed when the model declared it done within the limits and a separate check of the final page confirmed every condition: an accepted final URL, the task’s facts visible on the page, more than 100 characters of page text, and a fully loaded page.
Each attempt had 900 seconds, 60 browser actions and 120 decision calls. API errors counted as failures. Timing runs from the first loaded page to the final check.
The models could not see the page-loading status the check required. Jev on B05 and Gemini on M04 reached the right content but failed only that check, and they count as failures.
Cost
API cost covers the decision calls and the text helper, for all 50 attempts per model. OpenRouter reported the fees for GPT-6 Astra, Gemini and Mercury. Jev’s cost is its reported tokens at the list price on 22 September 2026: $0.042 per million input tokens and no charge for output.4 GLiDE, Clef and Clef-Flash are priced at their list prices too.
Self-hosted browser runs used an A100, A6000, RTX 4090 or a Mac. We price all of them on the A100 from the classification cost test, at $1.673 an hour: each model’s cost per input token there, times its browser input tokens, plus the text helper’s fees.
Both workloads are almost all input, because these models return option scores or a few tokens. The estimate assumes cost grows in proportion to input tokens. Browser inputs averaged 2 to 12 times the length of the same model’s classification questions, and we did not run the browser pages through the throughput test.
Fine-tuning
The 327 tasks, 286 for training and 41 held out, come from eight websites that are not in the benchmark, including Demo Web Shop, Automation Exercise and Project Gutenberg. Each task was checked in a browser: its start page fails the grader, and where the task ends on a different URL, that page passes.
GLM-5.3-Flash, an open-weight model under the MIT license,16 attempted 300 of the tasks on the benchmark runtime at low reasoning. Provider errors kept it from the other 27. We did not use GPT-6 Astra or Gemini for this, because OpenAI’s and Google’s terms bar using their output to build competing models.1718
The 184 successful training runs gave 606 decision records. A run counted as successful when the grader passed it, or when it reached the right product page through site search and failed only the exact-URL check, as 11 runs across all tasks did.
The single-decision test replays 103 recorded decisions from 19 held-out tasks and counts how often a model picks the labeling model’s operation. Kev could read 101 of them. Laya could read 88 when sent only the questions each step needs, and 34 in the full request the benchmark sends. On those 34, stock Laya agreed 29% of the time, the first fine-tune 71% and the second 41%.
Both models started from the stock weights we benchmarked and used their producers’ training recipes with the benchmark’s input limits. Kev also trained for two epochs instead of one, without the producer’s replay records. Each training run took under 20 minutes on one H100 GPU.
Laya’s first fine-tune used the same 606 records, skipping any too long for Laya. The second kept only the 292 that fit the full request the benchmark sends and added 102 from Laya’s own runs, where GLM-5.3-Flash took over after Laya declared a task done or blocked. Kev had no second round.
The Kev pair and Laya’s first pair ran on a rented H100 in France, with Chrome on Linux. Laya’s second pair ran on the Mac from the main benchmark. Every pair used the main benchmark’s 50 tasks, runtime, limits and grader, with one attempt per task.
Classification benchmark
Each item is one decision with a fixed set of options. Items come from records published in July to September 2026 and from unpublished snapshots of final pages in our own browser runs. Training overlap cannot be ruled out.
The first three tasks ask a model to reproduce a decision an institution recorded, from the text alone. GitHub and the FDA may have used evidence the text does not contain. GitHub’s severity levels are the four levels of CVSS, the standard vulnerability scoring system.19
Every model received the same instructions and option definitions for a task: the FDA’s class definitions, GitHub’s severity bands, arXiv’s category descriptions. Option order was shuffled per item and was the same for every model. Security flaw severity and recall danger level are the main tasks, fixed before any run; paper topics and the task done check are supplementary.
The score is balanced accuracy: the success rate within each possible answer, averaged over the answers.
A model that always gives the most common answer scores 25 on severity, 33 on recall and 5 on paper topics. The chart score is the average of the two main tasks, where that answer scores about 29. Intervals are 95% bootstrap intervals that resample related items together: advisories for the same package, recalls from the same recall event, pages from the same browser task.
Two LLMs, Qwen 3.8 Max and Claude Opus 5.5, labelled a sample of 350 test items, 100 per task and 50 browser pages, without seeing the recorded answer.
They agreed with each other on 62 of 100 advisories, 69 of 100 recalls, 84 of 100 papers and 49 of 50 browser pages. Where they disagreed with the record on advisories and recalls, they were mostly one level off, toward the middle levels. They matched GitHub’s severity on 55 and 56 of 100 advisories.
Both judges gave the same answer, different from the record, on 54 items. We checked the 53 GitHub, FDA and arXiv records against the live source, and all matched, so no label changed.
The one browser page was labelled done because the grader counted at least 40 reviews on the full page, while the snapshot the models see shows about nine. We removed it and the 48 other task done items whose answer depended on counting 10 or more items, which leaves 155.
Local costs hold for the setting we tested on that GPU; another server setting or GPU would change them. Between the scored run and the cost run, CLM-v0.1-8B changed 3.7% of its answers, Shisa DE-1 2.2% and Tev1 0.8B 1.2%, so their costs come from a setup that did not exactly reproduce their scores. The other 14 local models changed under 1%.
Exceptions to running each model as released:
- Laya ran with an 8,192-token context instead of its released 512 (1,024 for typed-decisions), and Tev1 with 8,194 instead of 2,050, as in our browser runs.
- Where Laya’s own code would have cut a long option or input, our connector rejected the request instead, and a rejection counts as a wrong answer. Every paper-topic request was rejected because the topic descriptions are longer than the 48 tokens Laya reads per option.
- CLM’s encoder context was raised from 2,048 to 16,384 tokens; GLiNER ran beyond its documented 512-token input. Shisa used a 32,768-token context to fit GPU memory.
- We could not run Decision-1.0-Lux-9B: its runtime runs this model only on AMD (ROCm) GPUs, and we used an NVIDIA GPU.
- Together AI had not published a license for the Tev1 weights by 30 September 2026.
- Cloudflare’s service reads only the first 2,048 tokens of the item text for Clef and Clef-Flash, although the models accept 65,536. The 12 longer items, 9 on severity and 3 on recalls, count as wrong answers.
Limitations
Each model attempted each browser task once, so a gap of a few tasks between two models does not show which is better.
The 50 tasks come from nine websites, and tasks on the same site share page layouts. The models ran on live websites, which can change between runs.
The step format, an operation list plus target lists, was first built for Jev. Every model received the same format. We did not test whether another format would change the results. Laya ran with input limits above its released settings so that long pages were not cut off.
In the classification benchmark, each model answered each item once. The GitHub and FDA answers are those institutions’ recorded decisions, which may rest on evidence the notice or recall text does not contain.
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{kalelioglu2026,
author = {Kalelioğlu, Berk},
title = {{AIM-Decision: Jev vs Kev vs LLMs}},
year = {2026},
month = oct,
howpublished = {\url{https://aimultiple.com/decision-models}},
note = {AIMultiple. Retrieved October 5, 2026}
}Results and timestamps of 257 data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 10 CSV files and a README.
Want the granular data behind it? Join Premium
Reference Links
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.