DecisionBench: Jev vs Kev vs LLMs
Decision models, also called System One models,1 choose an agent’s next action in a single pass instead of generating text token by token. To see whether they can make browser automation cheaper than LLMs, we ran three decision models and two LLMs, Gemini 3.8 Flash and GPT-6 Astra, on the same 50 browser tasks, for 250 attempts in total.
Decision model success rates
Kev-9B completed 20 of the 50 tasks and Jev 1.13 completed 17. Laya completed none in our integration, which ran above its released input limits.
We then ran the same 50 tasks with an LLM choosing each action. GPT-6 Astra completed 47 and Gemini 3.8 Flash 42, both at the low reasoning-effort setting of their API. Astra also completed every task that Jev or Kev completed.
See 10 example tasks and every model’s result on Hugging Face, and how we ran the benchmark under methodology.
Decision models vs LLMs
Most LLMs write their answer one token at a time, and every token costs compute and time. For most browser steps the agent only picks an operation and a control on the page.
A decision model reads the page and the options, scores each option, and returns the most likely one. It returns a probability for each option instead of generated text, so a step needs no long output. If decision models become accurate enough, the same automation could run much faster and at a fraction of LLM cost.
The three decision models cannot write text, so a separate text model, Mercury 2.5,2 writes the values typed into form fields. We call it the text helper, and it was available to every model.
We ran all five models on browser-use’s open-source jev-ultrafast runtime,3 which was built for Jev. The three decision models are Jev 1.13 from TypeSafe,4 Kev-9B5 and Laya typed-decisions.6 GPT-6 Astra serves as a high-end LLM reference and Gemini 3.8 Flash as a low-cost one.
Five models on the same task
On Wikipedia, every model except Laya typed into the search box, the text helper wrote “Ada Lovelace”, and the model picked the right suggestion. Jev reached the article in 8.1 seconds, Gemini in 11.0, Astra in 18.9 and Kev in 43.8. Laya declared the task done on the main page without taking an action.
The filter task needs two dropdowns and a search button. Jev, Gemini and Astra took the same four steps and finished in 4.6, 10.7 and 20.3 seconds. Laya clicked the site logo and declared the task done. Kev failed this task in the benchmark, and in the demo it was still clicking after 11 minutes, so it is left out of this clip.
Captions show each model’s chosen operation and target, and for decision models the probability of that operation. The clips come from a separate demo run at 1× speed, not from the scored benchmark, and each final page was checked with the benchmark’s grader.
Decision model results
The tasks Jev and Kev both completed include Wikipedia lookups, a dropdown selection, product sorting and a product tab. Both completed the two Wikipedia tasks and the Web Scraper dropdown task, as did both LLMs.
Each Books to Scrape task required moving from a category list to a product page. Jev completed none of the six and Kev three.
Together, Jev and Kev completed 26 distinct tasks.
Failure patterns
- Jev declared the task blocked in 24 of its 33 failures, 16 times before taking any action.
- Kev took a median of 9.5 actions on failed tasks, against 3 on tasks it completed. Ten attempts hit the action or decision-call limit, and five more were stopped after three actions in a row left the page unchanged.
- Laya declared the task finished before meeting the goal in 33 attempts, 17 times before taking any action. Eight more attempts were stopped after three unchanged actions. Nine attempts failed when our connector, the code that passes each request to Laya, rejected a question or option longer than Laya’s input limits.
Why Laya scored 0/50
We tested two changes to Laya’s integration. Across 24 test requests, sending each question separately instead of in one batch produced identical outputs. With a separate yes/no question for “is the goal already met?”, Laya passed at most 3 of 12 scenarios that required the right action first and a done signal after.
In all 12 scenarios, Laya scored the finished page higher than the unfinished one, but the score ranges overlapped from one scenario to the next. Eight of its 11 wrong targets repeated the value or page already on screen.
The Laya version we used was trained for judgment tasks such as support tickets, invoices and agent-trace review, not for browser navigation.6 We did not train any model further for browser tasks.
LLM results
GPT-6 Astra failed 3 tasks. It marked a ScrapeMe category task as blocked. On a calculator form it kept typing into the same field and was stopped after three actions left the page unchanged. The third ended in a text-helper error.
Gemini 3.8 Flash failed 8. Four of those reached the correct page but did not finish. Two were stopped by our cost safeguard when OpenRouter did not report a fee, one hit the action limit, and one ended in a text-helper error.
Both LLMs failed form task T02. Both filled and submitted the form, and both attempts ended when the text helper returned no valid value.
Success by website
Jev and Kev were strong on different sites. Kev completed 6 of 8 Web Scraper tasks and 3 of 6 on Books to Scrape, where Jev completed none. Jev did better on Quotes to Scrape (4 of 8) and Scrape This Site (3 of 6).
TestPages had all four form tasks and the fewest completions. Kev completed one and Jev none. For comparison, GPT-6 Astra completed every task on seven of the nine websites and two of the four form tasks.
Each website has two to eight tasks, so these counts apply only to this task set.
Speed and success
Jev was the fastest, at a median of 4.2 seconds per attempt, but part of that speed comes from stopping early. Sixteen of its failures ended before any action. On the 17 tasks Jev completed, it averaged 5.4 seconds against 12.5 for GPT-6 Astra on the same tasks.
Gemini (12.7 seconds, 84%) and Astra (17.4 seconds, 94%) were both fast and accurate. Laya took 15.1 seconds per attempt and completed no tasks.
Kev took a median of 38.3 seconds per attempt, for two reasons. Each Kev decision took a median of 3.7 seconds of GPU time, against about 1 second for a full Jev API call. We ran Kev with the producer’s reference code in full 32-bit precision, and two optional kernel packages that speed up its base model were not installed, so the model fell back to slower reference kernels. The network added about half a second.
Kev also made far more decisions, an average of 26 per attempt against about 4 for Jev, and 90% of them came on failed attempts. We did not test a faster GPU setup.
Aggiungi come fonte preferitaCost
Astra, Gemini and Jev each used 3.150 to 3.470 input tokens per decision. So the price per token, not the amount read, sets the cost gap. Astra’s input price is about 240 times Jev’s.78
At low reasoning, Astra produced 6.599 output tokens across 307 decisions, against 968.376 input tokens. Jev’s output price is $0.
At Jev’s input price, Astra’s 968.376 decision input tokens would have cost about $0,04 instead of $9.68. A decision model that matched Astra’s success rate at Jev’s price would run the same automation for a small fraction of the LLM cost.
Kev and Laya had no decision API fee. The A100 rental for Kev cost an estimated $5,31 (3.32 hours at $1,60 per hour), including setup time outside the scored runs. We did not estimate a cost for the Mac that ran Laya. One Jev fee and two Gemini fees were not recorded and are missing from the table.
Methodology
All five models used the same jev-ultrafast browser runtime. At each step, it turned the current page into page text plus a numbered list of visible controls. The model chose an operation, such as click, type, select, done, or blocked, and a target from that list. The runtime carried out the action and re-read the page.
The three decision models rank supplied options and cannot write text. When any model chose to type, the text helper wrote the value into the form field.
Each attempt started in a fresh browser session. All five models ran on the same day with the same tasks, runtime and grading.
Models
Low reasoning is a provider-defined effort setting. Astra’s 307 decisions included 791 reported reasoning tokens.
Dataset
The 50 tasks cover nine websites: eight practice sites built for scraping and testing, plus Wikipedia. The tasks include catalog navigation, search, sorting, pagination, product options, forms and article lookups. Each task starts from a fixed URL and states the end state in one sentence, for example “Switch to Science Fiction, find Dune (Dune #1), open its product details, and stop.”
Before the models ran, a scripted run completed all 50 tasks using only the runtime’s supported actions. Every task is solvable without passwords, CAPTCHAs or the Enter key.
Example tasks
Ten of the 50 tasks are published on Hugging Face with the instructions, start URL, pass conditions and every model’s result. The other 40 are withheld for later runs.
Evaluation
A task passed when the model declared it done within the limits and a separate check of the final page confirmed every condition: an accepted final URL, the task’s facts visible on the page, more than 100 characters of page text, and a fully loaded page.
Each attempt had 900 seconds, 60 browser actions and 120 decision calls. API errors counted as failures. Timing runs from the first loaded page to the final check.
The models could not see the page-loading status the check required. Jev on B05 and Gemini on M04 reached the right content but failed only that check, and they count as failures.
Cost
API spend is the decision calls plus the text helper, for all 50 attempts per model. OpenRouter reported the fees for GPT-6 Astra, Gemini and Mercury. Jev’s cost is its reported tokens at the list price on 22 September 2026: $0,042 per million input tokens and no charge for output.4 The Kev estimate has not been matched to an invoice.
Limitations
Each model attempted each task once, so the results do not estimate reliability. Kev completed 20 tasks and Jev 17; one run cannot say whether that difference would hold.
The 50 tasks come from nine websites, and tasks on the same site share page layouts. The models ran one after another on the same day, and live sites can change between runs.
The step format, an operation list plus target lists, was first built for Jev. Every model received the same format. We did not test whether another format would change the results. Laya ran with input limits above its released settings so that long pages were not cut off.
Cita questo benchmark
Scegli il formato adatto a dove pubblicherai. Incollare la versione con link nel tuo CMS preserva il backlink.
@misc{kalelioglu2026,
author = {Kalelioğlu, Berk},
title = {{DecisionBench: Jev vs Kev vs LLMs}},
year = {2026},
month = sep,
howpublished = {\url{https://aimultiple.com/decision-models}},
note = {AIMultiple. Consultato il 23 settembre 2026}
}Collegamenti di riferimento
Sii il primo a commentare
Il tuo indirizzo email non verrà pubblicato. Tutti i campi sono obbligatori. I commenti vengono lasciati nella loro lingua originale.