AIM-Code Bench: Agentic Coding Benchmark

Evaluating LLMs without their coding-agent harnesses does not fully reflect how they are used in practice. Real development also involves extending existing work, taking over unfamiliar code, and leaving implementations that others can build on. We introduce AIM-Code Bench to evaluate models together with their coding-agent harnesses across evolving software tasks. Beyond correctness, the benchmark measures runtime performance, adaptation to changing requirements, and outcomes when agents build on one another’s code.

Results

Fable 5.1 had the lowest observed latency in stages 1 and 2; Opus 5 led stages 3 and 4.

The table shows p95 latency (the 95th percentile) in milliseconds; lower is better. Each value is the median of ten replay measurements of the saved program. Ranks use full precision within each stage. The same stage-4 measurements serve as own-code controls in the handover comparison. Close numerical ranks do not necessarily indicate distinguishable performance.

Part 1: developing and improving their own implementations

Each agent kept its code and conversation across stages. We updated the task specification as new requirements arrived and gave it up to 20 minutes for each stage. The prompts described ordinary development work and asked agents to pursue the best performance they could achieve while preserving the required behavior.

The experiment covered thirteen model-and-product configurations: four in Claude Code, three in Codex, four in OpenCode, one in Grok Build and one in Devin CLI. Effort was medium where supported, high for Kimi K3 and GLM 5.3, and the provider default for MiniMax M3. Each result therefore belongs to the model and coding product together.

We tried to reflect as closely as possible how people actually use AI coding products in real-life scenarios. Many people use models within their own harnesses; therefore, while we utilized the available models within their respective harnesses, we used open-weights models via opencode.

Four stages of the project

Stages 1–3 used 5,000 operations per batch. Stage 4 used independent HTTP requests with 1,000 operations each, with one CPU and 128 MiB of memory allocated to the candidate service.

MiniMax M3’s measured batch latency went from 37.833 ms in the initial stage to 12.254 ms in the capacity stage. This is an observed reduction of 67.6% as the implementation changed and eviction became a requirement. Its original stage-4 HTTP implementation later ranked 4th, at 3.009 ms.

Qwen 3.8 Max’s measured batch latency moved from 11.638 to 8.023 ms between the first two stages, an observed 31.1% reduction. Its stage-3 result ranked 4th. These trajectories show why a single initial result misses changes that appear as an agent continues developing a project.

These percentages describe the recorded trajectories, not controlled optimization speedups. Each stage changes the required behavior or workload. In particular, stage-4 latency cannot be subtracted from stage-3 latency: the interface and operations per request differ. A direct before/after optimization claim would require replaying both versions under the same applicable workload.

How much code changed along the way?

The four stages cover 52 verified original-stage diffs across the thirteen configurations. Stage 4 illustrates the variation in implementation size and editing style:

These counts include agent-written source tests and tools, comments and blank lines; supplied task/test updates and JSON diagnostics are excluded. They measure the final difference from each stage’s starting snapshot. Larger changes can include useful tests or optimizations; smaller changes are not automatically better. The linked diff report contains the per-stage source changes.

Part 2: taking over another agent’s implementation

After measuring the original trajectories, we asked two different questions: whose code gives other agents a useful starting point, and which agent performs best when extending someone else’s code?

We branched from the saved stage-3 implementations and asked recipients to perform stage 4. Each of the thirteen configurations received the other twelve configurations’ code, giving 156 ordered donor–recipient pairs. There were no new self-takeovers. Every recipient started a fresh isolated session with the donor code and the same stage-4 requirements; it could not see the harness or other agents’ sessions.

The original stage-4 submissions serve as own-code controls, using the same measurements as Part 1. Those controls had been developed with the original conversation history, while the takeovers began fresh. The comparison therefore combines starting-code and conversation effects. This transition asks successors to extend an existing cache into an HTTP service while optimizing request latency.

View A: benefit supplied by a donor’s code

For each pair, we compare the successor’s HTTP latency with that same successor’s own original stage-4 latency:

successor benefit = (recipient own latency − inherited-code latency) / recipient own latency

A donor’s score is the median of these twelve percentage benefits. Positive means its successors typically achieved lower latency than their own original continuations. This is a performance-based view of how useful a starting implementation was; it does not directly measure documentation quality or ease of understanding.

Opus 5 supplied the largest median successor benefit at +17.30%, followed by Opus 5.5 at +15.70% and Fable 5.1 at +11.34%. Those outcomes can reflect the inherited implementation’s performance as well as how easily successors could extend it. Percentages normalize scale, but do not eliminate starting-quality or optimization-headroom differences.

View B: performance as a successor

For each donor, we calculate the median latency achieved by its twelve recipients. Every recipient of that donor is then compared with this shared reference:

successor advantage = (same-donor median latency − recipient latency) / same-donor median latency

The recipient’s score is its median advantage across twelve donors. Positive means it was typically faster than the median recipient of the same starting code. This avoids rewarding a recipient simply because its own original implementation was slow.

GPT-6 Astra led this exploratory successor summary at +2.68%, followed by Opus 5 at +2.33% and Grok 4.6 at +0.88%. Astra’s lead over Opus 5 was not distinguishable within the measured uncertainty. Being the strongest donor and being the strongest successor are different outcomes in this experiment.

Did successors build on the inherited source?

We report source churn separately from performance: (added lines + deleted lines) / inherited source lines. We also count handovers deleting more than 50% of inherited source. This is a large-deletion rate, not proof of a rewrite from scratch: renames, reformatting and removal of auxiliary code can cross the threshold.

None of the selected successors deleted over half of the total inherited source from GPT-6 Astra, Claude Fable 5.1, Claude Opus 5, Claude Opus 5.5 or Qwen 3.8 Max. All twelve crossed that threshold for Kimi K3, GPT-5.6 Luna, MiniMax M3 and Claude Sonnet 5. These are observations about physical-line changes, not a maintainability ranking. Donor programs contained only 75–668 source lines including auxiliary code, and the choice to include tests/tools affects deletion percentages.

Measurement method, retries and scope

Each reported performance score uses ten replays of the same saved program. Stages 1–3 used 40 measured batches per replay. Stage 4 and handover timing used 1,200 HTTP requests per replay at 20 requests per second for 60 seconds, with one CPU, 128 MiB memory and a 64-process/thread limit for each candidate.

We measured programs serially on one dedicated Intel i7-7700 Linux server. The driver and candidate ran on separate physical cores with unused sibling threads, a fixed 2 GHz CPU policy and turbo disabled.

All 156 selected handover submissions and all 52 original-stage builds passed every replay. The complete evaluation contains 2,080 measurements, including all thirteen own-code controls. We checked raw responses, recomputed p95 latency, verified source hashes and resource limits, and retained every scheduled measurement.

We calculate 95% intervals from 10,000 resamplings of the ten measurement rounds, using the same sampled rounds across programs. A comparison is unresolved when its difference interval includes zero. These exploratory intervals are not adjusted for multiple comparisons. Counts of faster successors compare measured scores and do not imply statistically distinguishable differences.

Across 13 original stage-4 builds and 156 handovers, all 2,028,000 measured HTTP responses were correct. Maximum dispatch delay was 0.607 ms. Every replay met the driver limits of p95 dispatch delay below 0.05 ms and maximum delay below 1 ms. Delay remains included in latency.

The published selection includes additional attempts. GLM 5.3 and Qwen 3.8 Max use their second complete original trajectories after initial stage-3 failures. Four completed handover pilot builds were reused, fifteen Claude pairs received infrastructure recovery attempts, and seven failed handovers received one fresh coding attempt from their original donors. All seven coding retries passed. New attempts were selected regardless of outcome; earlier attempts remain archived. These results compare selected builds, not first-attempt reliability.

The evaluator distinguishes candidate failures from infrastructure-invalid measurements. HTTP correctness requires at least 99% of requests served, correct warmups and no incorrect responses. Each ranked build must have ten valid replays. No code changed during the timing evaluation.

This is one project and one selected trajectory per configuration. The four stages are related, and the 156 handover pairs share donors and recipients. The intervals describe replay variation on this host; they do not cover other tasks, machines or independent coding runs. The findings describe these artifacts under these workloads, rather than a general ranking of coding ability, code comprehension or production maintainability.

Don’t miss our benchmarks and data-driven insights. The button opens Google; selecting AIMultiple confirms that you wish to see AIMultiple more often in Google search results.

Add as preferred source

What is next?

The next experiments will add more projects and independent coding trajectories, raise the difficulty of stages that reach the offered-load ceiling, and measure applicable before/after versions under the same workload. Fresh-session self-continuation controls would help separate the effects of conversation history from starting with another agent’s code.

Cite this benchmark

Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.

@misc{alper2026,
  author = {Alper, Şevval},
  title  = {{AIM-Code Bench: Agentic Coding Benchmark}},
  year   = {2026},
  month  = sep,
  howpublished    = {\url{https://aimultiple.com/agentic-llm}},
  note   = {AIMultiple. Retrieved September 22, 2026}
}

Changelog

13 updates

All13General9Methodology4

2026

1. Jul 23

   Added Kimi K3 and Inkling model results to the benchmark leaderboard.

2. Jul 23

   Added Kimi K3 and Inkling to the agentic coding benchmark lineup.

3. Jul 10

   Added Grok 4.5 and the GPT 5.6 model family to the coding benchmark comparison.

4. Jul 10

   Added seven models to the benchmark: Grok 4.5 and the GPT-5.6 family (Sol, Terra, Luna).

5. Jul 2

   Added comparison of code-specialist models GPT 5.3 Codex and Kimi K2.7 Code to backend scoring section.

6. Jul 1

   Added Claude Sonnet 5 to the benchmark leaderboard with a 0.772 overall score.

7. Jun 15

   Added Kimi K2.7 Code and Claude Fable 5 to the benchmark results and cost comparison sections.

8. Jun 15

   Updated the harness methodology: Claude Fable 5 ran on the Claude Code CLI subscription instead of Opencode via OpenRouter.

9. Jun 3

   Added Claude Opus 4.8, Gemini 3.5 Flash, and MiniMax M3 to the benchmark leaderboard and cost comparisons.

10. May 20

    Replaced the benchmarked model lineup and all success, cost and latency results with new values.

11. May 20

    Replaced Aider with opencode as the agent harness, and added three runs per cell to measure variance.

12. Apr 3

    Replaced Gemini 3 Pro with Gemini 3.1 Pro Preview in the benchmark results summary.

13. Feb 24

    Added Gemini 3.1 Pro Preview to the Agentic LLM benchmark results.

Şevval Alper

AI Researcher

Follow On

Şevval is an AI researcher at AIMultiple. She has previous research experience in pseudorandom number generation using chaotic systems.
Şevval focuses on AI coding tools, AI agents, and quantum technologies.

View Full Profile

Be the first to comment

Your email address will not be published. All fields are required. Comments are left in their original language.

Name

Email Address

Comment

0/450

Post Comment