GPT-6 Astra added
The newest model in the AIM-Visual-Reasoning benchmark is GPT-6 Astra.
AIM LMM Index
v1.0Large multimodal models (LMMs) answer questions about images, charts and video. The AIM LMM Index combines AIMultiple's visual reasoning benchmark with four public leaderboards. How the index works
BEST MODEL
GPT-6 Astra
Index 82 · OpenAI
BEST OPEN-SOURCE MODEL
Qwen3.8 27B
Index 71 · Alibaba Cloud
BEST ON AIM-VISUAL-REASONING
Gemini 4 Argon
74.5% correct · Google
AIM LMM Index vs. cost
Higher is more accurate, further left is cheaper. Cost is what the 200-question AIM-Visual-Reasoning run cost through each provider's API.
Leaderboard
Models with a score on at least 3 of the five benchmarks, ranked by the AIM LMM Index.
Filter & Sort
# | Model | AIM LMM Index | AIM-Visual-Reasoning | MMMU-Pro | Roboflow Vision | MedXpertQA MM | CAIS Vision | Run cost (USD) | Seconds per question |
|---|---|---|---|---|---|---|---|---|---|
| 1 |  GPT-6 Astra OpenAI | 82 | 71.5 | 87.5 | 83.4 | 86.8 | 83.0 | 7.30 | 17.9 |
| 1 |  Gemini 4 Argon Google | 82 | 74.5 | - | 87.5 | - | 83.3 | 2.92 | 16.7 |
| 3 |  Claude Opus 5.5 Anthropic | 81 | 73.5 | 87.4 | 82.0 | 85.7 | 76.8 | 18.51 | 43.2 |
| 3 | GPT-6.1 Sol OpenAI | 81 | 72.5 | 85.8 | 81.7 | 85.7 | 77.9 | 1.46 | 15.1 |
| 5 | Gemini 3.8 Flash Google | 79 | 73.0 | 85.3 | 79.8 | 84.7 | 69.9 | 1.34 | 13.6 |
| 5 | Claude Sonnet 5.5 Anthropic | 79 | 70.0 | 85.8 | 77.6 | 80.7 | - | 17.14 | 61.3 |
| 5 | Gemini 3.6 Flash Google | 79 | 70.5 | 83.7 | 77.1 | 82.8 | - | 1.37 | 8.8 |
| 8 | GPT-6 Sol OpenAI | 78 | 72.0 | 83.0 | 76.9 | 81.3 | - | 2.38 | 14.5 |
| 8 | Gemini 3.7 Flash Google | 78 | 73.5 | 85.2 | 80.1 | 83.5 | 68.7 | 0.86 | 8.5 |
| 8 |  Qwen3.8 Max Alibaba Cloud | 78 | 72.0 | 82.8 | 79.3 | 78.3 | - | 4.11 | 73.7 |
Page 1 of 4
Rank#1
AIM-Visual-Reasoning71.5
MMMU-Pro87.5
AIM-Visual-Reasoning
71.5
MMMU-Pro
87.5
Roboflow Vision
83.4
MedXpertQA MM
86.8
CAIS Vision
83
Rank#1
AIM-Visual-Reasoning74.5
AIM-Visual-Reasoning
74.5
Roboflow Vision
87.5
CAIS Vision
83.3
Rank#3
AIM-Visual-Reasoning73.5
MMMU-Pro87.4
AIM-Visual-Reasoning
73.5
MMMU-Pro
87.4
Roboflow Vision
82
MedXpertQA MM
85.7
CAIS Vision
76.8
Rank#3
AIM-Visual-Reasoning72.5
MMMU-Pro85.8
AIM-Visual-Reasoning
72.5
MMMU-Pro
85.8
Roboflow Vision
81.7
MedXpertQA MM
85.7
CAIS Vision
77.9
Rank#5
AIM-Visual-Reasoning73
MMMU-Pro85.3
AIM-Visual-Reasoning
73
MMMU-Pro
85.3
Roboflow Vision
79.8
MedXpertQA MM
84.7
CAIS Vision
69.9
Rank#5
AIM-Visual-Reasoning70
MMMU-Pro85.8
AIM-Visual-Reasoning
70
MMMU-Pro
85.8
Roboflow Vision
77.6
MedXpertQA MM
80.7
Rank#5
AIM-Visual-Reasoning70.5
MMMU-Pro83.7
AIM-Visual-Reasoning
70.5
MMMU-Pro
83.7
Roboflow Vision
77.1
MedXpertQA MM
82.8
Rank#8
AIM-Visual-Reasoning72
MMMU-Pro83
AIM-Visual-Reasoning
72
MMMU-Pro
83
Roboflow Vision
76.9
MedXpertQA MM
81.3
Rank#8
AIM-Visual-Reasoning73.5
MMMU-Pro85.2
AIM-Visual-Reasoning
73.5
MMMU-Pro
85.2
Roboflow Vision
80.1
MedXpertQA MM
83.5
CAIS Vision
68.7
Rank#8
AIM-Visual-Reasoning72
MMMU-Pro82.8
AIM-Visual-Reasoning
72
MMMU-Pro
82.8
Roboflow Vision
79.3
MedXpertQA MM
78.3
Page 1 of 4
Score on each benchmark
Each row shows a model's index and its published score on each benchmark. Hover a point to see all of the model's scores.
Published score (0-100)
How the index works
The AIM LMM Index combines AIMultiple's visual reasoning benchmark with four public leaderboards that test models on images, charts and medical scans.
Scoring
The index is the plain average of a model's published scores, taken as each leaderboard reports them, with no rescaling. The average covers only the benchmarks the model has a score on. The benchmarks differ in difficulty: most CAIS Vision scores are below 70 while most MMMU-Pro scores are above 70, so models with the same set of benchmarks compare most directly.
Which models are ranked
A model is ranked when it has a score on at least 3 of the five benchmarks. Only the models in AIMultiple's visual reasoning benchmark are listed.
Public scores
When a leaderboard lists a model at several reasoning levels, we use the highest one (max, then xhigh, then high). Runs that use tools, such as Python, are left out. CAIS contributes its Vision Capabilities Index average.
Benchmarks we used
AIMultiple
AIM-Visual-Reasoning40 models on 200 questions about charts and images, scored on overall accuracy.Public
MMMU-ProCollege-level questions across disciplines that need both the image and the text. Mercor APEX leaderboard, pass@1; Artificial Analysis for models Mercor has not tested.Public
Roboflow Vision EvalsReal-world images across five tasks: object detection, counting, identification, OCR and visual reasoning. Roboflow Playground overall score, the average of the five tasks.Public
MedXpertQA MMExpert-level medical questions that pair clinical text with images. Mercor APEX leaderboard, pass@1.Public
CAIS Vision IndexThe Center for AI Safety's average over five vision tests: embodied reasoning, spatial navigation, mental visualization, intuitive physics and puzzles.Changelog
4 updates- v1.0
33 models entered the index: Claude Fable 5, Claude Fable 5.1, Claude Haiku 4.5, Claude Haiku 5.5, Claude Opus 5 and 28 more.
- v1.0
Added 7 models to the benchmark, not ranked in the index yet: Claude Opus 4.6, Claude Sonnet 4.5, GPT-5.1-Codex, GPT-5.2 Pro, Kimi K2.5 and 2 more.
- v1.0
AIM-Visual-Reasoning: Replaced the max tokens parameter setting in the Benchmark methodology.
Explore Large Multimodal Models
Large Multimodal Models (LMMs) vs LLMs
LMMInsightOct 20Evaluate LLMs and LMMs by comparing their benchmark scores and real-world latency by clicking the model’s name in the table below. You can also weigh their input and output pricing to judge overall efficiency and value. *Audio is native on the E2B, E4B and 12B models only. Available in five sizes: E2B, E4B, 12B, 26B…
- LMMBenchmarkOct 20
Compare Multimodal AI Models on Visual Reasoning
We benchmarked 40 leading multimodal AI models on visual reasoning using 200 visual-based questions. The evaluation consisted of two tracks: 100 chart understanding questions testing data visualization interpretation, and 100 visual logic questions assessing pattern recognition and spatial reasoning. See our benchmark methodology to learn our testing procedures. Every model scores lower on visual logic…
Filters