Şevval Alper

Sep 22

AIM-Code Bench: Agentic Coding Benchmark

Evaluating LLMs without their coding-agent harnesses does not fully reflect how they are used in practice. Real development also involves extending existing work, taking over unfamiliar code, and leaving implementations that others can build on. We introduce AIM-Code Bench to evaluate models together with their coding-agent harnesses across evolving software tasks. Beyond correctness, the benchmark…

Sep 22

AI Code Review Tools Benchmark

With the increased use of AI coding tools, codebases have become more prone to vulnerabilities, which increased the need for effective code reviews. To address this, we introduce RevEval (AI Code Review Eval), which benchmarks the top four AI code review tools across 309 pull requests from repositories of varying sizes and evaluates their performance…

Sep 21

Agent benchmarks

Agent benchmark

Sep 18

Agentic IT: Can AI Agents Design a Benchmark

We tested 16 models on a benchmark design in text-to-SQL and tool calling. Each model built one benchmark per topic, for a total of 32 submissions. No submission passed every rubric criterion. The agents could build and run tests, but none demonstrated both a blank-answer test and a correct-answer test of their own scorer. Text-to-SQL…

Sep 16

AI Coding Benchmark: Claude Code vs Cursor

In AI coding, the market has fragmented into two categories: Agentic CLI tools and AI code editors embedded in IDEs. Each claims to automate development. Few comparisons show how they differ under identical workloads. We benchmarked each agent across 10 full-stack web development tasks, performing ~600 atomic validation checks per agent and more than 9,600…

VELC-Bench: Verification on Long Context Benchmark

The model’s ability to locate a specific metric in context, compare its value to a claim, and confirm or reject it. This tests fine-grained value matching under long-context conditions. The model must both retrieve the value and perform a precise comparison. The models are tested in the following context windows: claude-fable-5 scores 90.0% on verify…

RELC-Bench: Retrieval on Long Context Benchmark

RELC-Bench (RELC-Bench: Retrieval on Long Context Benchmark) aims to measure a model’s ability to find and extract a specific numeric value from one or more documents within its context. It tests whether the model can remember and retrieve a specific fact it just saw in the input. claude-fable-5 scores 97.0% on the 100 direct-recall items,…

Audience Simulation: Can LLMs Predict Human Behavior?

In marketing, evaluating how accurately LLMs predict human behavior is crucial for assessing their effectiveness in anticipating audience needs and recognizing the risks of misalignment, ineffective communication, or unintended influence. Audience simulation with LLMs enables the modeling of virtual audiences, helping organizations anticipate reactions to content or products without relying on costly surveys or focus…

HALC-Bench: LLM Hallucination on Long-Context Retrieval Benchmark

HALC-Bench (LLM Hallucination on Long-Context Retrieval Benchmark) measures a large language model’s resistance to fabricating evidence for a metric that does not exist in the target document by using 3 haystacks placed at the beginning, middle, and end of the model’s context window, with 204 questions. claude-fable-5 answered all 204 traps correctly at every haystack…