AIM-Code Bench: Agentic Coding Benchmark

Evaluating LLMs without their coding-agent harnesses does not fully reflect how they are used in practice. Real development also involves extending existing work, taking over unfamiliar code, and leaving implementations that others can build on. We introduce AIM-Code Bench to evaluate models together with their coding-agent harnesses across evolving software tasks. Beyond correctness, the benchmark measures runtime performance, adaptation to changing requirements, and outcomes when agents build on one another’s code.

Results

Opus 5 had the lowest observed latency in stages 1 and 3; Fable 5.1 led stages 2 and 4. Those are the leaders of the original development evaluation.

The table shows p95 latency in milliseconds; lower is better. Each value is the median of three replay measurements of the saved program. Ranks are calculated within each stage, using full precision. Later handover measurements use their own remeasured controls and are presented separately.

Part 1: developing and improving their own implementations

Each agent kept its code and conversation across stages. We updated the task specification as new requirements arrived and gave it up to 20 minutes for each stage. The prompts described ordinary development work and asked agents to pursue the best performance they could achieve while preserving the required behavior.

The experiment covered twelve model-and-product configurations: three in Claude Code, three in Codex, four in OpenCode, one in Grok Build and one in Devin CLI. Effort was medium where supported, high for Kimi K3 and GLM 5.3, and the provider default for MiniMax M3. Each result therefore belongs to the model and coding product together.

We tried to reflect as closely as possible how people actually use AI coding products in real-life scenarios. Many people use models within their own harnesses; therefore, while we utilized the available models within their respective harnesses, we used open-weights models via opencode.

Four stages of the project

Stages 1–3 used 5,000 operations per batch. Stage 4 used independent HTTP requests with 1,000 operations each, with one CPU and 128 MiB of memory allocated to the candidate service.

MiniMax M3’s measured batch latency went from 12.053 ms in the initial stage to 2.526 ms in the capacity stage. This is an observed reduction of 79.0% as the implementation changed and eviction became a requirement. Its original stage-4 HTTP implementation later ranked 2nd, at 0.908 ms.

Qwen 3.8 Max’s measured batch latency moved from 2.496 to 2.096 ms between the first two stages, an observed 16.1% reduction. Its stage-3 result ranked 3rd. These trajectories show why a single initial result misses changes that appear as an agent continues developing a project.

These percentages describe the recorded trajectories, not controlled optimization speedups. Each stage changes the required behavior or workload. In particular, stage-4 latency cannot be subtracted from stage-3 latency: the interface and operations per request differ. A direct before/after optimization claim would require replaying both versions under the same applicable workload.

How much code changed along the way?

The four stages cover 48 verified original-stage diffs across the twelve configurations. Stage 4 illustrates the variation in implementation size and editing style:

These counts include agent-written source tests and tools, comments and blank lines; supplied task/test updates and JSON diagnostics are excluded. They measure the final difference from each stage’s starting snapshot. Larger changes can include useful tests or optimizations; smaller changes are not automatically better. The linked diff report contains the per-stage source changes.

Part 2: taking over another agent’s implementation

After measuring the original trajectories, we asked two different questions: whose code gives other agents a useful starting point, and which agent performs best when extending someone else’s code?

We branched from the saved stage-3 implementations and asked recipients to perform stage 4. Each of the twelve configurations received the other eleven configurations’ code, giving 132 ordered donor–recipient pairs. There were no new self-takeovers. Every recipient started a fresh isolated session with the donor code and the same stage-4 requirements; it could not see the harness or other agents’ sessions.

The original stage-4 submissions were remeasured as own-code controls. Those controls had been developed with the original conversation history, while the takeovers began fresh. The comparison therefore combines starting-code and conversation effects. This transition asks successors to extend an existing cache into an HTTP service while optimizing request latency.

View A: benefit supplied by a donor’s code

For each pair, we compare the successor’s HTTP latency with that same successor’s own original stage-4 latency:

successor benefit = (recipient own latency − inherited-code latency) / recipient own latency

A donor’s score is the median of these eleven percentage benefits. Positive means its successors typically achieved lower latency than their own original continuations. This is a performance-based view of how useful a starting implementation was; it does not directly measure documentation quality or ease of understanding.

Opus 5 supplied the largest median successor benefit at +13.16%, followed by Fable 5.1 at +11.19% and SWE-2 at +5.78%. Those outcomes can reflect the inherited implementation’s performance as well as how easily successors could extend it. Percentages normalize scale, but do not eliminate starting-quality or optimization-headroom differences.

View B: performance as a successor

For each donor, we calculate the median latency achieved by its eleven recipients. Every recipient of that donor is then compared with this shared reference:

successor advantage = (same-donor median latency − recipient latency) / same-donor median latency

The recipient’s score is its median advantage across eleven donors. Positive means it was typically faster than the median recipient of the same starting code. This avoids rewarding a recipient simply because its own original implementation was slow.

GPT-6 Astra led this exploratory successor summary at +3.81%, followed by GLM 5.3 at +3.34% and SWE-2 at +2.93%. Being the strongest donor and being the strongest successor are different outcomes in this experiment. The median-based formula can produce exact zero ties, as it does for MiniMax M3and Qwen 3.8 Max; those are not tolerance-based groupings of nearby latency values.

Did successors build on the inherited source?

We report source churn separately from performance: (added lines + deleted lines) / inherited source lines. We also count handovers deleting more than 50% of inherited source. This is a large-deletion rate, not proof of a rewrite from scratch: renames, reformatting and removal of auxiliary code can cross the threshold.

None of the selected successors deleted over half of the total inherited source from GPT-6 Astra, Claude Fable 5.1, Claude Opus 5 or Qwen 3.8 Max. All eleven crossed that threshold for Kimi K3, GPT-5.6 Luna, MiniMax M3 and Claude Sonnet 5. These are observations about physical-line changes, not a maintainability ranking. Donor programs contained only 75–668 source lines including auxiliary code, and the choice to include tests/tools affects deletion percentages.

Measurement method, retries and scope

Each reported performance score uses three replays of the same saved program. Handover timing used 240 HTTP requests per replay at 20 requests per second, with one CPU, 128 MiB memory and a 64-process/thread limit for each candidate.

All 132 selected handover submissions have valid passing results, and all twelve own-code controls passed. The latest full timing batch supplied 129 cross measurements and all controls. Three invalid cross measurements were recovered in later timing-only batches using unchanged code and settings; one needed another recovery after a further dispatch spike. Every valid result from the full batch was retained. No invalid partial score was used.

Across the final 103,680 measured HTTP requests, maximum retained dispatch delay was 16.111 ms; 13 delays exceeded 5 ms, and none exceeded the original 25 ms validity guard. Delay remains included in latency. Timing noise can change close ranks, particularly across batches.

The published selection includes additional attempts. GLM 5.3 and Qwen 3.8 Max use their second complete original trajectories after initial stage-3 failures. Four completed handover pilot builds were reused, fifteen Claude pairs received infrastructure recovery attempts, and seven failed handovers received one fresh coding attempt from their original donors. All seven coding retries passed. New attempts were selected regardless of outcome; earlier attempts remain archived. These results compare selected builds, not first-attempt reliability.

The measurement pipeline also received a timing correction before the reported original evaluation and a startup-failure classification fix before the latest full handover evaluation. The latter distinguishes a confirmed candidate exit from the Docker DNS error that follows it. Original correctness requirements and resource limits were retained; the audit trail is linked below.

This is one project and one selected trajectory per configuration. The four stages are related, and the 132 handover pairs share donors and recipients. The findings describe these artifacts under these workloads, rather than a general ranking of coding ability, code comprehension or production maintainability.

Don’t miss our benchmarks and data-driven insights. The button opens Google; selecting AIMultiple confirms that you wish to see AIMultiple more often in Google search results.

Add as preferred source

What is next?

The next experiments will add more projects and independent coding trajectories, raise the difficulty of stages that reach the offered-load ceiling, and measure applicable before/after versions under the same workload. Fresh-session self-continuation controls would help separate the effects of conversation history from starting with another agent’s code.

Cite this benchmark

Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.

@misc{alper2026,
  author = {Alper, Şevval},
  title  = {{AIM-Code Bench: Agentic Coding Benchmark}},
  year   = {2026},
  month  = sep,
  howpublished    = {\url{https://aimultiple.com/agentic-llm}},
  note   = {AIMultiple. Retrieved September 22, 2026}
}

Changelog

13 updates

All13General9Methodology4

2026

1. Jul 23

   Added Kimi K3 and Inkling model results to the benchmark leaderboard.

2. Jul 23

   Added Kimi K3 and Inkling to the agentic coding benchmark lineup.

3. Jul 10

   Added Grok 4.5 and the GPT 5.6 model family to the coding benchmark comparison.

4. Jul 10

   Added seven models to the benchmark: Grok 4.5 and the GPT-5.6 family (Sol, Terra, Luna).

5. Jul 2

   Added comparison of code-specialist models GPT 5.3 Codex and Kimi K2.7 Code to backend scoring section.

6. Jul 1

   Added Claude Sonnet 5 to the benchmark leaderboard with a 0.772 overall score.

7. Jun 15

   Added Kimi K2.7 Code and Claude Fable 5 to the benchmark results and cost comparison sections.

8. Jun 15

   Updated the harness methodology: Claude Fable 5 ran on the Claude Code CLI subscription instead of Opencode via OpenRouter.

9. Jun 3

   Added Claude Opus 4.8, Gemini 3.5 Flash, and MiniMax M3 to the benchmark leaderboard and cost comparisons.

10. May 20

    Replaced the benchmarked model lineup and all success, cost and latency results with new values.

11. May 20

    Replaced Aider with opencode as the agent harness, and added three runs per cell to measure variance.

12. Apr 3

    Replaced Gemini 3 Pro with Gemini 3.1 Pro Preview in the benchmark results summary.

13. Feb 24

    Added Gemini 3.1 Pro Preview to the Agentic LLM benchmark results.

Şevval Alper

AI Researcher

Follow On

Şevval is an AI researcher at AIMultiple. She has previous research experience in pseudorandom number generation using chaotic systems.
Şevval focuses on AI coding tools, AI agents, and quantum technologies.

View Full Profile

Be the first to comment

Your email address will not be published. All fields are required. Comments are left in their original language.

Name

Email Address

Comment

0/450

Post Comment