Agentic IT: Can AI Agents Design a Benchmark
We tested 16 models on a benchmark design in text-to-SQL and tool calling. Each model built one benchmark per topic, for a total of 32 submissions. No submission passed every rubric criterion. The agents could build and run tests, but none demonstrated both a blank-answer test and a correct-answer test of their own scorer.
Benchmark design scores
Text-to-SQL turns a plain-language question into a database query. Tool calling selects a function and fills in its arguments.
We graded each submission against a rubric the agents did not see. The text-to-SQL rubric has 78 points, and the tool-calling rubric has 74. Scores show the percentage of applicable rubric points. Bar labels are rounded to whole numbers.
- Text-to-SQL: Claude Opus 5 and Grok 4.6 tie at 78.2%. When we resample the rubric criteria, the middle 95% of their ranks fall between 1st and 5th.
- Tool calling: GPT 5.6 Sol leads at 74.3%. The middle 95% of its resampled ranks fall between 1st and 4th. The rank spans come from resampling the rubric criteria.
Across the 16 models, the two topic scores correlate at 0.42 (Pearson). The average score combines two rubrics with different criteria, so it is not a like-for-like comparison.
Missing quality checks
No submission passed these four checks:
- Blank-answer test: Empty answers go through the scorer. A working scorer should award close to zero points.
- Correct-answer test: The reference answers go through the full extraction and scoring steps. A working scorer should award full marks.
- Missing-reference test: The cases are repeated without the database schema or tool definitions.
- Difficulty target: The strongest model should score between 40% and 60%. It scored above 60% in 30 of 32 submissions.
Twenty submissions also missed the required gap of at least 20 points between the strongest and weakest models.
- Calibration: Two submissions passed the calibration check: GLM 5.3 in text-to-SQL and Claude Opus 5 in tool calling. To pass, a submission had to keep its revisions and explain them after missing the difficulty target.
- Predictions: Six of 32 submissions passed the prediction check. A pass required naming the best and worst models and placing at least two of their four scores inside the predicted intervals. Each agent also predicted the score interval and rank of a fifth, held-out model. Two submissions got both right: Gemini 3.8 Flash and Grok 4.6, both in text-to-SQL.
- Unsupported claims: Twenty-two of 32 reports failed the judge panel’s check for claims without supporting evidence.
Cost and time
Cost records cover 16 models and elapsed-time records cover 15. The chart uses the 15 with both records, omitting Grok 4.6 because its duration is unrecorded.
Costs cover the authoring agent’s inference per topic, including recorded retries. They exclude API calls made by the generated benchmark programs, so they understate the complete cost of building and running a benchmark.
The runs use different agent programs and include replacement attempts. The data cannot show whether more spending or longer runs lead to higher scores.
The score intervals obtained by resampling rubric criteria average 32.7 points wide. They exclude the variation that repeated authoring runs would add.
What the models got right
Every result row in the 32 main submissions links to a saved model response. Re-running each submission’s scorer on those saved responses reproduced its result table. The scoring is repeatable, but reproduction does not authenticate the saved responses.
We separately deleted and corrupted sampled response files. All 32 scorers changed their output and told a deleted response apart from a corrupted one. The test does not check every scoring rule.
Twenty-two of 32 reports failed the panel’s check for unsupported claims.
What the results mean for agentic IT
In IT operations, an AI agent reads the state of a system and acts on it, for example by installing a patch or restarting a service. A separate check then has to confirm that the action worked. The scorer control tests in this benchmark play the same role for grading: they confirm the scorer works before its results are used. None of the 32 main submissions passed those tests.
This benchmark covers text-to-SQL and tool calling. It is not a test of patching, ticket triage, or other IT tasks.
We test IT management tools separately, on live systems. Selected results:
Add as preferred sourceAgentic IT platforms
The 16 models in this benchmark are LLMs. The platforms below are not. They sit above such models and add IT data, workflows, and controls.
An agentic system decides its next step from the state it observes, while rule-based automation follows steps set in advance. IT products combine the two, so each entry below notes which part is which.
Creatio
Creatio runs AI agents inside a CRM and workflow platform, so an agent acts on the company’s own records rather than on a chat transcript.
Models: OpenAI, Azure OpenAI, or any provider supported by the LiteLLM library, including models on the company’s own servers. Each agent can use a different one.
Teams build their own agents: An agent is defined in the interface, not in code: its instructions, then what it must not do, which data it cannot reach, which skills it can call, and which actions it can take on a record.
Coding agents build the apps: Claude Code, Codex, and GitHub Copilot connect to the platform through a plugin, agent skills, and MCP tools, and can create data models, pages, business rules, and test data.
ServiceNow
ServiceNow is an IT service management (ITSM) platform.
Models: Agent orchestration can run on Azure OpenAI, Claude on AWS, Google Gemini, or Now LLM, ServiceNow’s own model.
Confirmation before action: ITSM agents can be started by hand from the Now Assist panel, or they can run automatically when a record is created or updated. Admins decide which actions need a person to confirm them.
NinjaOne
NinjaOne is an endpoint management platform covering monitoring, patching, backup, and remote access.
Models: NinjaOne does not document a model choice for its AI features. They run as part of the platform, unlike the options above, where an admin picks the provider.
Patch risk scored from community reports: Patch Intelligence AI reviews vendor telemetry and public reports from other administrators about Windows updates, then flags updates that broke systems elsewhere and summarizes the context. The other platforms in this section read a company’s own records; this feature reads what happened at other companies.
CVE detection without a scan: The vulnerability module identifies CVEs from software telemetry analyzed in NinjaOne’s cloud, so no scan runs on the endpoint, and findings pass to the patching module for remediation.
Methodology
The task
For each topic, every model received the same fixed prompt. The prompt asked the agent to:
- choose a business subject,
- write at least 24 test cases across four categories,
- write a runner (the program that sends cases to models) and a scorer (the program that grades answers),
- run four named models twice on each case.
A fifth model was held out. The agent predicted its results first and then tested it.
The prompt asked each agent to decide which quality standards a benchmark needs before publication and to show that it met them. The prompt did not state the numeric difficulty target or name any scorer test.
Agents and submissions
The submissions ran between July and September 2026. Most used opencode, an agent program that gives the model a shell and a file system.
The previous version covered 14 models and 28 submissions. Our historical data also include seven older models that are no longer in the roster.
We also ran five pilot setups for GPT 5.5, resulting in 10 additional submissions. Their prompts differ from the main task, so the charts exclude them. In total, this update includes 42 submissions and scores 40. Two pilot submissions did not meet the minimum dataset size, so the judge panel did not grade them.
Grading
Code checks the submitted files, rebuilds databases, re-runs scorers and matches result rows to response files. A model panel grades the criteria that need interpretation. Submitted code runs in an isolated copy with no network access.
We exclude a live re-execution of sampled model calls for every submission because the runners have different interfaces and some lack usable provider configurations. Without that criterion, the rubrics total 78 points for text-to-SQL and 74 for tool calling.
All 32 submissions in the charts pass the required file and dataset-size checks. Task prompts were not edited for this update, and the results use one retained submission per model and topic.
Scorer control tests
The missing-reference test checks whether models can answer without the information they are supposed to need. If they still score well, the questions may reveal the answer, or the models may have seen the data during training. A high score alone is not proof of either.
One pilot prompt named all three scorer tests, and that pilot ran them in both topics. Re-running its saved test files with no network access reproduced the recorded numbers.
The prompts for the 32 main submissions left these tests to the agent, and none of those submissions passed any of the three. A single prompt in the study named the tests, so their effect across models remains untested.
The judge panel
GPT 5.6 Sol and Claude Opus 5 grade every criterion sent to the panel. When they disagree, Grok 4.6 casts the deciding vote.
The judges run at temperature 0 (the least random setting), with high reasoning effort and a 32,000-token limit. If a judge returns incomplete or invalid JSON, no verdict is recorded and the call is retried.
Across 672 criteria, pilots included, the two main judges disagreed on 195. Sol passed at 54.8%, and Opus at 82.6%. Grok passed 133 of the disputed criteria.
The judges’ vendors also have graded submissions in this benchmark: four from OpenAI, four from Anthropic, and one from xAI. The judge prompts leave out author names and remove file paths that could identify a model. Writing style may still reveal the author.
On submissions from vendors other than OpenAI and Anthropic, Sol scores 13.9 points below Opus. On OpenAI submissions, the gap was 9.1 points, or 4.8 points smaller. On Anthropic-authored submissions, Opus’s lead over Sol is 0.1 points smaller than on other vendors’ work. Pilot prompts differ, so the audit does not establish favoritism. Grok judges only disputes, including disputes over its own submissions, and its votes are outside this audit.
Intervals
The score intervals and middle 95% of ranking positions use 4,000 paired draws of rubric criteria with replacement. Each submission is rescored on the same drawn criteria as the others in its topic. The intervals describe dependence on the rubric and exclude task-sampling and run-to-run uncertainty. Correlations are Pearson.
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{kalelioglu2026,
author = {Kalelioğlu, Berk and Alper, Şevval},
title = {{Agentic IT: Can AI Agents Design a Benchmark}},
year = {2026},
month = sep,
howpublished = {\url{https://aimultiple.com/agentic-it}},
note = {AIMultiple. Retrieved September 18, 2026}
}Results and timestamps of 21 data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 2 CSV files and a README.
Want the granular data behind it? Join Premium
Şevval focuses on AI coding tools, AI agents, and quantum technologies.
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.