Agentic IT: Can LLMs Design a Benchmark
We gave 12 large language models the job a benchmark team does: invent a benchmark, build it, run four models through it, and report the results. Each did it twice. None of the 24 attempts passed every criterion, and six of the rubric’s checks were passed by none of them.
Benchmark design scores
The two topics are text-to-SQL, turning a question into a database query, and tool calling, picking the right function and arguments from a set on offer. Each attempt was graded against a rubric the models never saw, worth 78 points for the first and 74 for the second.
The chart carries the 12 models this campaign runs across every one of its benchmarks, so a model’s place here can be read against its place in the others. Four more attempted this task and are in the published data file without a chart position; the methodology gives the counts.
On rank range, the scores separate one model and no others. Claude Opus 5 leads text-to-SQL at 78.2 and Kimi K3 leads tool calling at 74.3, and their rank ranges overlap those of every other charted model but one. A rank range is the span of finishing positions a model holds across 4.000 bootstrap draws. Inkling Small is the exception, and it comes last in text-to-SQL in all 4.000.
The two topics agree on most of the field, with scores correlating at 0.82. They do not agree at the top. Opus 5 leads text-to-SQL and places fifth in tool calling, while K3 goes from third to first.
A 78.2 means the attempt earned %78,2 of the rubric points that applied to it. The two rubrics contain different criteria, so scores compare within a topic and not across.
Missing quality checks
Six checks have no passes across the 24 attempts, and three of them are the ones that test whether a benchmark’s own scorer works.
The first runs blank answers through the scorer. It should award close to nothing, and one that hands out points for empty submissions is measuring its own bugs. The second is the opposite: feed in the correct answers and the score should come back %100.
The third takes the database schema or the tool list away and asks the questions again. If the models still answer, they knew the answers already and the benchmark is measuring memory.
We searched all 222 scripts written across every run we scored, pilots included, first by file name and then by content. These three tests turn up in exactly two runs, both from the same pilot configuration, and that configuration’s prompt listed them by name. Re-running its files in a sandbox with no network reproduced its reported numbers: blank answers scored %0, correct answers %100.
Not one unprompted attempt ran any of the three, 24 out of 24, across every vendor in the field. The only prompt that named them got them back. The gap sits in the habit rather than in the instruction.
The difficulty miss is the failure most likely to reach a published leaderboard unnoticed. The rubric asks the strongest model under test to land between %40 and %60, where a benchmark can still discriminate in both directions. All 24 attempts built something easier, and 17 of them also failed to open a 20-point gap between the best and worst model they tested.
Missing the band on the first try is forgivable. The task then asks for a rebuild and the rounds it took, and none of the 24 supplied them.
Most models cannot predict their own benchmark’s results. Each attempt wrote down, before running anything, the pass rate it expected from each of the four models it would test. In 14 of 24, fewer than two of the four landed inside the range the attempt had set. Counting those that also named the best or worst model wrong, 18 of 24 fail.
They do no better on a model they have never seen. The task holds a fifth model back: once the benchmark is finished on the first four, the attempt has to predict that model’s score and its placing before running it. None got both right.
One attempt came close on the memorization test. Opus 5’s tool-calling run ships a script that reruns the root-cause analysis behind one section of its report, and the panel judged it short of the four cases across all four models the criterion asks for.
Cost and time
Cost is the whole run divided by the topics attempted, failed attempts included, so it reads as what one benchmark build cost. Scores here are the mean of a model’s two attempts. Switch the x-axis to elapsed time.
Price and time are the same finding here. Cost and elapsed time correlate at 0.85, because the expensive models are the slow ones. Score correlates with cost at 0.45 and with time at 0.53.
Hold time constant and the price effect disappears. Among models that run about as long as each other, the partial correlation between cost and score is 0.00, while time keeps 0.32. The money buys a model that thinks for longer, and little else.
What it does buy is tidiness rather than validity. The six checks nothing passed carry 15 rubric points per attempt, and the five most expensive models and the five cheapest both earn zero of the 150 available to them. Their whole gap sits in the other points, %87 against %76.
None of these gaps is separated in the statistical sense. The %95 intervals behind the scores average 33 points, wider than the spread of the whole field.
What the models got right
No attempt invented a result row. Every result row across the 24 attempts has a stored provider response behind it, and all 24 scorers reproduce their reported numbers exactly when re-run on that stored data.
That reproduction is not a hardcoded table. We damaged five stored responses in each attempt and re-ran its scorer: all 24 changed their results, and all 24 told a deleted response apart from a corrupted one.
The write-up is a different matter. The rubric asks that no claim in the report go beyond what the results file supports, and 15 of the 24 attempts fail that check.
Tercih edilen kaynak olarak ekleMethodology
The task
Each model received the same prompt: invent a domain, build a dataset of at least 24 cases across four categories, write a runner and a scorer, then call four named models twice on every case. A fifth model is held back, and the attempt has to predict its result before running it.
The prompt names the categories and the goal. It names none of the difficulty band, the scorer tests or the borrowed-content checks the rubric applies.
Harness and runs
The article reports the 12 models this campaign runs across every one of its benchmarks, two attempts each, so a model’s standing here can be carried to the other two. Every number above is one of those 24 attempts. Seventeen models were given the task and sixteen delivered; the four outside the shared set keep their rows in the published data file with the reason they are not charted.
Every attempt ran through the opencode agent harness, one isolated working directory per attempt, models routed through OpenRouter, under a wall-clock limit. The 16 delivering models ran in two batches: eleven between 2026-07-10 and 2026-07-17, five on 2026-08-05.
The prompt text is unchanged across every revision on record. The five August models each log the version they received; the eleven July models predate that record, so their copy cannot be checked against it.
Ten further runs came from five pilot configurations of a single model, written while the task was being designed. One of the five was prompted with the scorer tests listed by name, which is why its two runs are the only ones that carry them. All ten sit outside the field, and counting them, 42 runs were scored.
Grading
Scoring splits in two. Deterministic criteria run against each attempt’s own artifacts in a sandbox: rebuild the database twice and compare, re-run the scorer and diff its output, execute every gold query, count stored responses against reported rows. Judgement criteria go to a panel.
One criterion, a live re-execution of sampled model calls, was excluded everywhere because the 42 runners share no common interface, which is why the denominators are 78 and 74 rather than 80 and 76.
Three runs are marked invalid for missing a file or a scoring step the rubric requires before a score means anything: one tool-calling attempt and two 12-case pilots that predate the 24-case requirement. All three sit outside the 12-model field, so no count above rests on a score that does not mean anything.
Qwen 3.8 Max was given the same task and produced no scorable attempt in either topic, so it appears in no number here.
The judge panel
The panel is three models from three vendors. GPT 5.6 Sol and Claude Opus 5 score every criterion, one call per criterion, seeing the criterion text and the artifacts but never whose work it is. Where the two disagree, Grok 4.5 answers that criterion and a majority decides.
The judges disagreed on 200 of 700 criteria. Opus was the passing vote in 195 of those, and Grok passed 152, so the tie-break went to the more generous reading three times in four. Across all 700, Sol passed %53,6 and Opus %80,7, which makes Sol the strict reader rather than the other two lenient.
Two of the three judges also appear as scored attempts. Against a baseline of how each judge scores work from neither vendor, Sol scores GPT-authored attempts 3.3 points higher and Opus scores Anthropic-authored attempts 0.2 points higher.
Intervals
The %95 intervals come from redrawing the rubric’s criteria 4.000 times and rescoring every attempt on the redrawn list, paired so a draw favours the same criteria for everyone. They measure how much a score depends on which criteria the rubric happens to contain, and each attempt is graded on 36 or 37 of them. Correlations are Pearson.
Bu araştırmayı kaynak gösterin
Yayınlayacağınız yere uygun formatı seçin. Bağlantılı sürümü CMS'inize yapıştırmak, geri bağlantıyı korur.
@misc{kalelioglu2026,
author = {Kalelioğlu, Berk and Alper, Şevval},
title = {{Agentic IT: Can LLMs Design a Benchmark}},
year = {2026},
month = aug,
howpublished = {\url{https://aimultiple.com/agentic-it}},
note = {AIMultiple. Erişim tarihi: 11 Ağustos 2026}
}
Yorum yapan ilk kişi olun
E-posta adresiniz yayınlanmayacak. Tüm alanlar gereklidir. Yorumlar orijinal dilinde bırakılır.