Benchmark of 40+ LLMs in Finance: Claude Fable 5 & GPT-5.6 Sol
We evaluated 40+ LLMs in finance on 238 hard questions from the FinanceReasoning benchmark to identify which models excel at complex financial reasoning tasks like statement analysis, forecasting, and ratio calculations.
LLM finance benchmark overview
We evaluated LLMs on 238 hard questions from the FinanceReasoning benchmark (Tang et al.).1 This subset targets the most challenging financial-reasoning tasks, assessing complex, multi-step quantitative reasoning involving financial concepts and formulas. Our evaluation employed a custom prompt design and scoring criteria of accuracy and token consumption.
For a detailed explanation of how these metrics were calculated and the framework used for this evaluation, please see our financial benchmark methodology.
Results: Which LLM is the best for finance?
Top-tier performers (>83% accuracy):
gpt-5.6-sol-pro reaches the benchmark’s highest accuracy at 90.76% with 385,886 tokens at $16.35 per run, 0.42 points above gpt-5.6-sol.
gpt-5.6-sol scores 90.34% accuracy with 117,735 tokens at $3.85 per run. It ties claude-fable-5 (90.34%) for the second-highest accuracy, at $3.85 against fable-5’s $10.05 and 117,735 tokens against 183,258. No model at or above its accuracy runs at a lower cost.
claude-fable-5 scores 90.34% accuracy with 183,258 tokens. It was the first model to pass 90% on this benchmark (June 10) and ties gpt-5.6-sol at 90.34%, at $10.05 per run against gpt-5.6-sol’s $3.85.
claude-opus-4.8 scores 89.08% accuracy with 113,434 tokens, the third-highest costed result and the lowest output token count among models above 88%.
gpt-5-2025-08-07 scores 88.23% accuracy with 829,720 tokens.
claude-opus-4.6 scores 87.82% accuracy with 164,369 tokens, near-top accuracy at 20% of gpt-5’s token count.
gpt-5-mini-2025-08-07 reaches 87.39% accuracy with 595,505 tokens.
gemini-3.5-flash reaches 86.97% accuracy with 1,191,757 tokens, the highest accuracy in Google’s Flash line and the family’s heaviest token user.
claude-sonnet-5 scores 86.97% accuracy with 414,668 tokens. It matches gemini-3.5-flash to the decimal while spending 414,668 tokens against 1,191,757, at $4.33 per run against $10.83.
gpt-5.6-terra scores 86.97% accuracy with 121,936 tokens at $1.99 per run. It joins claude-sonnet-5 and gemini-3.5-flash at 86.97%, at the lowest token count and cost of the three (121,936 tokens and $1.99, against 414,668 / $4.33 and 1,191,757 / $10.83). gpt-5-mini-2025-08-07 stays cheaper and more accurate ($1.21, 87.39%).
gemini-3.1-pro-preview scores 86.55% accuracy with 475,148 tokens, above its predecessor gemini-3-pro-preview (86.13%) at 35% fewer tokens (730,759 tokens).
glm-5.2 scores 86.13% accuracy with 735,988 tokens. It improves on glm-4.5 (64.29%) by 21.84 points, the largest jump between two versions of one model family in the benchmark. The next-largest is 3.79 points.
gemini-3-pro-preview and gpt-5.2 tie at 86.13% accuracy. gpt-5.2 reaches it at 247,660 tokens, about three times fewer output tokens than gemini-3-pro-preview at 730,759 tokens.
claude-opus-4.7 scores 85.29% accuracy with 103,268 tokens, the most token-efficient model in the top tier (37% fewer output tokens than claude-opus-4.6 while clearing the 83% threshold).
Strong performers (80-83% accuracy):
grok-4.3 reaches 84.87% accuracy with 309,781 tokens, xAI’s strongest result in the benchmark and a recovery from the earlier grok-4-0709.
claude-opus-4.5 scores 84.03% accuracy with 144,505 tokens.
claude-sonnet-4.6 and gemini-3-flash-preview tie at 83.61% accuracy. Claude Sonnet 4.6 uses 161,035 tokens, while Gemini 3 Flash Preview reaches it with 118,530 tokens, the lowest token count among models above 83%.
kimi-k2.5 scores 82.77% accuracy with 877,868 tokens, the highest consumption in this performance tier.
Middle tier (70-80% accuracy):
o3-pro-2025-06-10 (78.15% accuracy, 473,659 tokens) and kimi-k2 (78.15% accuracy, 100,323 tokens) tie, and kimi-k2 uses 79% fewer tokens. o3-mini-2025-01-31 (77.31% accuracy, 376,929 tokens), gpt-5-nano-2025-08-07 (76.89% accuracy, 1,028,909 tokens), and claude-sonnet-4-20250514 (76.05% accuracy, 135,462 tokens) follow.
Low performers (<70% accuracy):
claude-3-5-sonnet-20241022 (67.65% accuracy, 90,103 tokens) and gpt-oss-20b (67.65% accuracy, 515,041 tokens) lead this tier. gemini-2.5-flash (65.55% accuracy, 286,603 tokens), glm-4.5 (64.29% accuracy, 692,662 tokens), and gpt-4.1-nano-2025-04-14 (63.45% accuracy, 171,096 tokens) follow.
Performance insights:
Token consumption does not track accuracy. deepseek-r1-0528 consumed the most tokens (1,251,064) at 62.18% accuracy, while claude-opus-4-20250514 scored 80.25% with 132,274 tokens. Token efficiency varies among high-accuracy models too: gemini-3-flash-preview reaches 83.61% with 118,530 tokens, while kimi-k2.5 spends 877,868 tokens for 82.77% (7.4x the tokens for 0.84 points less accuracy).
The table above presents other AI model benchmarks, including those utilized for this benchmark.
Cost per benchmark run
Output tokens alone do not determine spend, since input-token and output-token rates differ by an order of magnitude on most providers. We computed the dollar cost of running each model on the 238 questions using the provider’s listed per-token rate at the time of the run.
Lowest cost in the frontier tier: gemini-3-flash-preview reaches 83.61% accuracy for $0.39, the lowest dollar cost in the >83% accuracy band. grok-4.3 follows at $0.86 for 84.87%.
The 90.34% tier is reached at a lower cost: gpt-5.6-sol matches claude-fable-5’s 90.34% at $3.85 per run against fable-5’s $10.05, at 38% of fable-5’s cost. gpt-5.6-sol-pro records the benchmark’s highest accuracy, 90.76%, at $16.35 per run. claude-opus-4.8 stays the cheapest model to clear 88%, at $3.28 for 89.08%.
Premium reasoning at premium price: o1-pro is the highest-cost model in the benchmark at $381 for 80.67% accuracy (driven by $150/M input and $600/M output rates). o3-pro costs $39.23 for 78.15%, o1 costs $46.59 for 74.79%. None clears the top tier, and all three sit below claude-opus-4.7 on accuracy at more than 10x its cost.
Budget-tier cost and accuracy: gpt-oss-120b reaches 81.09% accuracy at $0.06 total, the cheapest run in the benchmark, within 1.91 points of the 83% threshold. llama-4-maverick reaches 75.21% at $0.10. For workloads where 80% accuracy is enough, these models cost less than 1% of the frontier flagships.
Financial reasoning benchmark methodology
Our benchmark provides a transparent and reproducible evaluation of LLM performance on complex financial reasoning tasks.
Test setup & data corpus
- Benchmark suite: We used the data, code, and evaluation scripts from the FinanceReasoning benchmark, selected for its focus on quantitative and inferential financial problems.
- Knowledge corpus & test queries: We focused our analysis on the hard subset, comprising 238 challenging questions. As defined by the benchmark, each data point includes:
- A question requiring multi-step logical and numerical deduction.
- A context, which often contains dense information presented in structured formats like Markdown tables (e.g., balance sheets, stock performance data).
- A definitive ground truth answer for objective scoring.
- Illustrative query types: The benchmark’s difficulty stems from its requirement for models to handle diverse and complex financial reasoning tasks. To illustrate this breadth, we highlight two representative examples from the test set:
Example: Algorithmic & time-series reasoning (technical analysis)
Context: An investor is analyzing… stock prices over the last 25 days… to calculate the Keltner Channel using a 10-day EMA period and a 10-day ATR period, with a multiplier of 1.5…
Question: What is the value of the last upper band in the Keltner Channel…? Answer to two decimal places.
This query tests a model’s ability to act as a quantitative analyst by:
- Deconstructing a composite indicator: Recognizing that the “Keltner Channel” is derived from two other complex indicators:
- The exponential moving average (EMA)
- The average true range (ATR).
- Implementing algorithmic logic: Correctly implementing the iterative algorithms for both EMA and ATR from scratch over a time series of 25 data points.
- Synthesizing results: Combining the calculated values according to the final Keltner Channel formula (Upper Band = EMA + (Multiplier × ATR)).
Core evaluation principles
- Isolated & standardized API calls: For each model, we conducted the evaluation programmatically via their respective API endpoints (e.g., OpenRouter, OpenAI). This ensured that every model received the exact same input under identical conditions, eliminating variability from UI interactions.
- Free-form generation: We did not constrain the models to a multiple-choice format. Instead, they were prompted to generate a comprehensive, free-form response, allowing for a more authentic assessment of their reasoning capabilities.
- Chain-of-Thought (CoT) prompting: To elicit and evaluate the models’ reasoning process, we employed a Chain-of-Thought (CoT) prompting strategy. The system prompt explicitly instructed each model to “first think through the problem step by step” before concluding with a final answer. This approach allows for a deeper analysis of how a model arrives at its conclusion, beyond the final output.
Evaluation metrics & framework
We utilized the FinanceReasoning benchmark’s own fully automated evaluation framework to score the model outputs. This framework is designed to measure both conceptual correctness and computational cost.
1. Primary metric: Accuracy
This metric answers the critical question: “Can the model correctly solve the financial problem?” The scoring process involves a sophisticated two-step pipeline:
- Step 1: LLM-based answer extraction: A model’s raw output is unstructured text containing both reasoning and a final answer. To reliably parse the numerical or boolean value, we used a supervisor model (anthropic/claude-sonnet-4.5) as the parser.
- Step 2: Tolerance-based comparison: A simple “exact match” is insufficient for numerical problems. Therefore, the extracted answer was programmatically compared against the ground truth. Script applies a numerical tolerance threshold (a relative difference of 0.2%) to fairly handle minor floating-point or rounding variations, ensuring that conceptually sound solutions are marked as correct.
2. Secondary metric: Token consumption
This metric answers the question: “How computationally expensive is it for the model to solve these problems?” It measures the total cost associated with generating the 238 answers.
- Token calculation: For each API call we collected prompt_tokens and completion_tokens from the provider’s usage object. The per-model token score is the sum of completion_tokens across all 238 questions. We report completion tokens (not total tokens) because input is nearly constant across models that share the same dataset (66k-92k input tokens per run depending on tokenizer).
- Cost calculation: We computed dollar cost as prompt_tokens × prompt_price_per_M + completion_tokens × completion_price_per_M, summed across all 238 questions. Prices are the listed per-token rates from the OpenRouter /api/v1/models endpoint at the time the model was run.
This two-metric approach, provided by the FinanceReasoning benchmark itself, allows for a holistic assessment, balancing a model’s raw problem-solving capability (accuracy) against its operational efficiency (token consumption).
Financial reasoning with Retrieval-Augmented Generation (RAG)
To surpass standalone models, we designed and implemented a custom RAG framework distinct from the benchmark’s original implementation. Our approach is built on a modern vector database stack (Qdrant) to supply LLMs with relevant, domain-specific knowledge at inference time, helping them solve problems beyond their training data. We tested this on gpt-4o-mini to measure its impact.
Results and analysis: The RAG trade-off
The introduction of RAG had a significant and measurable impact on the performance of gpt-4o-mini.
Key takeaways from the RAG evaluation:
- Significant accuracy improvement: RAG demonstrably enhanced the model’s problem-solving capability, boosting accuracy by over 10 percentage points. This confirms that providing external, relevant context is highly effective for complex, domain-specific reasoning tasks.
- The cost of accuracy: This performance gain came at a high cost. The total token consumption increased by nearly x18, and the total execution time increased by x20. This is due to the additional API calls for embedding and, more importantly, the vastly larger and more complex prompts that the LLM must process.
- Implications for larger models: The results from gpt-4o-mini suggest that while RAG can unlock higher performance, applying this method to larger, more expensive models like GPT-4o or Claude Opus will be substantially more costly and time-consuming. This highlights the critical trade-off between accuracy, cost, and latency in designing production-grade financial AI systems.
Financial reasoning RAG methodology
Our RAG pipeline is built on a modern stack using Qdrant as the vector database and OpenAI’s text-embedding-3-small model for generating semantic vector representations. The process consists of two main phases: an offline indexing phase and an online retrieval-generation phase.
1. Knowledge corpus indexing
- Corpus creation: We curated a specialized knowledge base from two sources provided by the benchmark:
- Financial documents: A collection of articles (financial_documents.json) explaining various financial concepts and terms.
- Financial functions: A library of ready-to-use Python functions (functions-article-all.json) designed to solve specific financial calculations.
- Intelligent chunking & embedding: To prepare this corpus for efficient retrieval, each document and function was processed and indexed:
- Chunking: Documents were segmented into smaller, semantically coherent chunks based on their sections. Each Python function was treated as a single atomic chunk. This ensures that the retrieved context is focused and relevant.
- Embedding: Each chunk was then converted into a 1536-dimension vector using the text-embedding-3-small model.
- Indexing: These vectors were indexed into two separate collections within our local Qdrant instance (financial_documents_openai_small and financial_functions_openai_small), optimized for cosine similarity search.
2. RAG-powered inference
For each of the 238 questions, the model’s reasoning process was augmented with the following automated steps:
- Embedding generation (API calls 1 & 2): The user’s query (question + context) was converted into an embedding vector. This required two calls to OpenAI’s embedding API to prepare for searches in both collections.
- Multi-source retrieval: The query vector was used to perform a semantic search against both Qdrant collections simultaneously to retrieve the most relevant information:
- The top 3 most relevant document chunks from the financial_documents collection.
- The top 2 most relevant Python functions from the financial_functions collection.
- Prompt augmentation: The retrieved documents and functions were dynamically injected into the prompt, creating a rich, context-aware “information packet”. This significantly increased the input prompt size (from ~300-500 tokens to ~3,000-5,000+ tokens).
- Final answer generation (API call 3): This augmented prompt was sent to the gpt-4o-mini model to generate the final, reasoned answer.
Add as preferred sourceLLMs in finance benchmark limitations
Our benchmark, while comprehensive, is subject to several key limitations:
- Data contamination risk: It is possible that these models have been trained on the benchmark’s dataset since the dataset is public. This could lead to inflated scores, making the true reasoning ability difficult to assess.
- Single-model RAG analysis: The RAG evaluation was performed on one model (gpt-4o-mini), so the observed trade-offs between performance and cost may not apply to all other models.
Conclusion
Our benchmark of 40+ models on complex financial reasoning tasks shows the following:
gpt-5.6-sol-pro reaches the benchmark’s highest accuracy at 90.76% with 385,886 tokens at $16.35 per run. gpt-5.6-sol and claude-fable-5 tie at 90.34%.
gpt-5.6-sol matches claude-fable-5’s 90.34% at $3.85 per run against fable-5’s $10.05, the lowest cost at the 90.34% accuracy tier.
claude-opus-4.8 scores 89.08% at 113,434 tokens for $3.28, above gpt-5-2025-08-07 (88.23%) at about 7x fewer output tokens and less than half the cost ($3.28 vs $8.38), and stays the cheapest model above 88% accuracy.
claude-opus-4.6 (87.82%, 164,369 tokens) and gpt-5-mini-2025-08-07 (87.39%, 595,505 tokens) reach near-top accuracy. gpt-5.6-terra reaches 86.97% at $1.99, the lowest cost among the three models tied at that accuracy.
gemini-3.1-pro-preview (86.55%) is above gemini-3-pro-preview (86.13%) at 35% fewer tokens, so iterative updates can improve both accuracy and efficiency.
gemini-3-flash-preview reaches 83.61% with 118,530 tokens at $0.39, and gpt-5.2 reaches 86.13% at 247,660 tokens.
RAG raised gpt-4o-mini’s accuracy by 10.08 points at a cost of 17.7x the tokens and 20x the latency.
Changelog
We add models to this benchmark with each new release.
July 10, 2026
- OpenAI: GPT-5.6 Sol (openai/gpt-5.6-sol)
- OpenAI: GPT-5.6 Terra (openai/gpt-5.6-terra)
- OpenAI: GPT-5.6 Sol Pro (openai/gpt-5.6-sol-pro)
June 30, 2026
- Anthropic: Claude Sonnet 5 (anthropic/claude-sonnet-5)
June 22, 2026
- Zhipu AI: GLM-5.2 (z-ai/glm-5.2)
June 10, 2026
- Anthropic: Claude Fable 5 (anthropic/claude-fable-5)
June 2, 2026
- Anthropic: Claude Opus 4.8 (anthropic/claude-opus-4.8)
May 22, 2026
- Google: Gemini 3.5 Flash (google/gemini-3.5-flash)
- xAI: Grok 4.3 (x-ai/grok-4.3)
- Added cost-per-benchmark-run column and the accuracy-vs-cost chart toggle. Methodology updated with the cost-calculation formula and the answer-extractor change (claude-sonnet-4.5)
April 20, 2026
- Anthropic: Claude Opus 4.7 (anthropic/claude-opus-4.7)
February 20, 2026
- Google: Gemini 3.1 Pro Preview (google/gemini-3.1-pro-preview)
- Anthropic: Claude Sonnet 4.6 (anthropic/claude-sonnet-4.6)
February 6, 2026
- Anthropic: Claude Opus 4.6 (anthropic/claude-opus-4.6)
- Anthropic: Claude Opus 4.5 (anthropic/claude-opus-4.5)
- Anthropic: Claude Sonnet 4.5 (anthropic/claude-sonnet-4.5)
- Google: Gemini 3 Pro Preview (google/gemini-3-pro-preview)
- Google: Gemini 3 Flash Preview (google/gemini-3-flash-preview)
- OpenAI: GPT-5.2 (openai/gpt-5.2)
- Moonshot AI: Kimi K2.5 (moonshotai/kimi-k2.5)
Further reading
Financial analysis may refer to multiple capabilities, such as stock analysis, financial law interpretation, and financial reasoning. In our benchmark, we focused specifically on financial reasoning, while other tasks are covered in separate articles:
- LLM for stock analysis: These models help process market data, company reports, and news to identify investment opportunities. (See full analysis here: AI-based Stock Trading)
- Finance law AI: Some LLMs can interpret financial regulations, contracts, and compliance requirements to assist legal-finance tasks. (See our legal AI tools list here: Legal AI Tools)
FAQs
An LLM (large language model) in finance is an AI model that uses natural language processing techniques to perform complex financial analysis, compliance management, and document understanding. These models help financial institutions navigate financial law, regulatory requirements, and the dynamic demands of the financial industry.
Intelligent chatbots:
LLM-driven virtual assistants enable financial firms to provide automated, 24/7 customer support by handling routine queries and onboarding tasks without human intervention. This reduces wait times and improves customer satisfaction while freeing human agents for complex issues.
Advisory & analysis:
Investment banks use LLMs to analyze market trends, financial news, and client data. These models digest large volumes of unstructured information, enabling advisors to deliver personalized investment advice and portfolio management with real-time insights.
Regulatory document analysis:
Law firms and financial institutions employ LLMs to process dense regulatory documents like SEC filings. These models extract key information and summarize reports, reducing manual review time and helping firms stay compliant with evolving regulations
Fraud detection:
LLMs analyze vast financial datasets in real time to detect suspicious transaction patterns and emerging fraud tactics. Their continual learning capabilities allow faster and more accurate fraud identification than traditional methods.
Legal and compliance automation:
Law firms and compliance teams use LLMs to review contracts, interpret banking laws, and verify regulatory compliance. Automating these tasks reduces review time and legal costs while ensuring adherence to complex financial regulations.
Document Q&A and Named Entity Recognition (NER):
Financial institutions deploy LLMs to answer questions from investors by extracting data from financial reports and earnings calls. NER enables automatic tagging of company names, stock tickers (class trading symbols), and regulatory entities, streamlining data retrieval.
Efficiency and automation: LLMs automate routine analysis (e.g., summarizing earnings reports, processing loans or filings), saving analyst hours and reducing errors.
24/7 customer service: AI virtual assistants and chatbots powered by LLMs can handle customer queries around the clock with conversational answers, improving customer experience and satisfaction.
Personalized financial advice: By analyzing a client’s history and risk profile, LLMs deliver tailored financial or investment advice.
Fraud detection & risk management: LLMs sift through large transaction datasets to spot anomalies or fraud patterns, adapting to new scam tactics and helping build risk profiles.
Compliance & reporting: LLMs automatically draft regulatory reports, extract policy-relevant facts, and help parse complex finance law and regulations for compliance.
Yes, several larger domain-specific models exist for finance. For example, BloombergGPT is designed to assist with financial regulation, capital markets, and compliance management by processing large financial datasets including documents from the national securities exchange and regulatory filings.
Other models like FinBERT and FinGPT focus on financial law, international banking law, and personalized financial advice, adapting large language models to the specialized vocabulary of finance such as class trading symbols and regulatory texts.
Financial reasoning is the ability to analyze financial data to make informed business or investment decisions.
Top tasks include:
– Analyzing financial statements (profit, cash flow, balance sheet)
– Budgeting and forecasting
– Evaluating investments (NPV, IRR, ROI)
– Managing cash flow and liquidity
– Assessing financial risks and performance ratios
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sar2026,
author = {Sarı, Ekrem},
title = {{Benchmark of 40+ LLMs in Finance: Claude Fable 5 & GPT-5.6 Sol}},
year = {2026},
month = jul,
howpublished = {\url{https://aimultiple.com/finance-llm}},
note = {AIMultiple. Retrieved July 10, 2026}
}Results and timestamps of 46 data points. Download the data used in this article as a ZIP file containing one CSV file and a README.
Reference Links
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.