LLM Benchmarks

ARC-AGI-1Few-shot abstract reasoning over grid transformations; the first ARC Prize benchmark.

ARC-AGI-2Novel visual reasoning puzzles testing generalization, not memorization.

ArxivMathRecent arXiv mathematics problems, run soon after publication to limit contamination.

BioMysteryBenchAnthropic's benchmark of whether AI agents can solve real bioinformatics mysteries from raw data.

CritPtResearch-level physics problems requiring multi-step derivation.

AIMultiple

FinanceReasoningComplex multi-step financial reasoning across 238 hard FinanceReasoning questions.

Frontier-BenchAgentic software tasks on frontier repositories, scored by resolution rate.

FrontierMathUnpublished, expert-authored research math problems (Tiers 1–4).

LegalBenchCrowd-sourced legal reasoning across 162 tasks from legal practitioners.

MMMU-ProCollege-level multimodal understanding across disciplines, hardened against text-only shortcuts.

Terminal-Bench 2.1Agentic terminal tasks across software engineering, data and system administration.

Terminal-Bench HardThe hard subset of Terminal-Bench: multi-step software tasks solved in a real terminal.

Terminal-Bench-ScienceTerminal tasks drawn from real scientific computing workflows.

AIMultiple

Text-to-SQLNatural-language-to-SQL query generation accuracy across 35+ LLMs.

Recent Updates

Latest changes to the Intelligence Index, model coverage and AIMultiple benchmark methodology.

Sep 2, 2026

Google

Gemini 3.8 Flash

New model added to the AIMultiple Intelligence Index.

Sep 1, 2026

Anthropic

Claude Fable 5.1

New model added to the AIMultiple Intelligence Index.

Aug 28, 2026

Tencent

Hy4 preview

New model added to the AIMultiple Intelligence Index.

Aug 26, 2026

Z AI

GLM 5.3 Flash

New model added to the AIMultiple Intelligence Index.

AI Model Release TimelineLast 12 models in 22 days

Key AI model releases from leading providers over the past 22 days.

OpenAI

04 Sep 2026

GPT-6 Astra +1 moreGPT-6 AstraGPT-6 Astra Pro

Google

02 Sep 2026

Gemini 3.8 Flash

Explore LLM Use Cases, Analyses & Benchmarks

Compare 9 Large Language Models in Healthcare

Feature Comparison

Sep 4

We benchmarked 9 LLMs using the MedQA dataset, a graduate-level clinical exam benchmark derived from USMLE questions. Each model answered the same multiple-choice clinical scenarios using a standardized prompt, enabling direct comparison of accuracy. We also recorded latency per question by dividing total runtime by the number of MedQA items completed. Benchmark methodology: This benchmark…

Sep 3

Large Multimodal Models (LMMs) vs LLMs

We evaluated the performance of Large Multimodal Models (LMMs) in financial reasoning tasks using a carefully selected dataset. By analyzing a subset of high-quality financial samples, we assess the models’ capabilities in processing and reasoning with multimodal data in the financial domain. The methodology section provides detailed insights into the dataset and evaluation framework employed.…

Sep 2

LLM Scaling Laws: Analysis from AI Researchers

Large language models predict the next token based on patterns learned from text data. The term LLM scaling laws refers to empirical regularities that link model performance to the amount of compute, training data, and model parameters used during training. To understand how these relationships influence modern model design in practice, we reviewed findings from…

Sep 2

LLM Observability Tools: Weights & Biases, Langsmith

LLM applications have expanded from single-turn chats into multi-step agents that use tools, query databases, and coordinate with other models, making their behavior harder to interpret. LLM observability provides continuous visibility into these complex workflows, helping organizations monitor quality, detect failures, troubleshoot issues, and manage performance and costs. W&B Weave is Weights & Biases‘ LLM…

Feature Comparison

Sep 2

Cloud LLM vs Local LLMs: Examples & Benefits

Cloud LLMs, powered by advanced models like GPT-5.5 and Claude Opus 4.7, offer scalability and accessibility. Conversely, Local LLMs, driven by open-source models such as Llama 4, DeepSeek V4, and Qwen3.6-Plus, ensure stronger privacy and customization. Explore what are cloud LLMs, strengths and weaknesses, most common case studies with real-life examples, and how they differ…

Benchmark

Audience Simulation: Can LLMs Predict Human Behavior?

In marketing, evaluating how accurately LLMs predict human behavior is crucial for assessing their effectiveness in anticipating audience needs and recognizing the risks of misalignment, ineffective communication, or unintended influence. Audience simulation with LLMs enables the modeling of virtual audiences, helping organizations anticipate reactions to content or products without relying on costly surveys or focus…

LLM Fine-Tuning Guide for Enterprises

Follow the links for the specific solutions to your LLM output challenges. If your LLM: The widespread adoption of large language models (LLMs) has improved our ability to process human language. However, their generic training often results in suboptimal performance for specific tasks. To overcome this limitation, fine-tuning methods are employed to tailor LLMs to…

LLM Market Share: Compare Usage & Adoption

We analyzed LLM market share by combining usage-based data and web visit estimates to show how demand for large language models is distributed across AI labs and AI applications: Read the methodology to see how we measured and calculated these results. The United States dominated web visits across all four months, consistently accounting for 85–95%.…

10+ Large Language Model Examples

We have gathered open-source benchmarks to compare leading proprietary and open-source large language models. Choose your use case to find the right model. You can evaluate large language models by examining their benchmark performance and real-world latency (available by clicking each model’s name in the table), and by reviewing their pricing to assess overall efficiency…

Aug 31

LLM VRAM Calculator for Self-Hosting

Self-hosting an LLM means running inference on hardware the operator controls rather than via a third-party API, which changes the cost, data control, and privacy profile. Whether a model runs at all depends on memory. The calculator estimates the VRAM or unified memory a model needs to run locally, based on the model, its precision,…