Artificial Intelligence
Explore practical insights, research, and benchmarks on artificial intelligence, including generative AI, large language models, RAG, governance frameworks, MLOps practices, and AI hardware. Gain an understanding of key tools, implementation strategies, and enterprise use cases shaping the AI landscape.
Explore Artificial Intelligence
LLM Orchestration: 22 Frameworks and Gateways
Optimizing LLM orchestration is key to improving performance while keeping resource use under control. To evaluate how different orchestration approaches perform in practice, we benchmarked: Discover selected LLM orchestration tools, including developer frameworks and enterprise gateways: LLM Orchestration involves managing and integrating multiple Large Language Models (LLMs) to perform complex tasks efficiently. It ensures smooth…
Cloud GPU Rental Price Index
On-demand rates for the newest-generation cloud GPUs (B200, B300, MI300X, RTX 5090) roughly doubled over the past year, while mainstream cards (H100, H200, A100) held a tight band. We compile the GPU index monthly from 69 providers and 17 GPU models, covering on-demand, spot, and 1-year reserved tiers. The chart shows the monthly median posted…
Handwriting Recognition Benchmark with 14 LLMs & OCRs
OCR tools achieve over 99% accuracy on typed text in high-quality images. However, handwriting remains challenging due to variations in style, spacing, and irregularities. We introduce a cursive handwriting recognition benchmark with 100 handwriting samples written by our team to prevent overfitting. In this benchmark, GPT-5, Gemini 3 Pro Preview, and olmOCR-2-7B-1025-FP8 are the top-performing…
AIM Enterprise: Agentic Enterprise Benchmark
Enterprises use LLMs every day for their regular tasks. To find the most cost-efficient LLMs, we designed AIM Enterprise, an agentic enterprise benchmark, where we used 69 real enterprise tasks across strategy, marketing, HR, sales, and operations. Two judge models scored every file, and they often disagreed. On about a third of the individual scores…
Text-to-SQL: Comparison of LLM Accuracy
We ran 36 large language models over 759 questions from BIRD-SQL, each model writing SQL against a database it had to identify for itself out of 11 candidates. Every parseable query was executed against the real database and its result set compared with the result set of BIRD’s gold query. Missing, malformed and execution-failing queries…
GPU Concurrency Benchmark: H100 vs H200 vs B200 vs MI300X
I have spent the last 20 years focusing on system-level computational performance optimization. We benchmarked the latest NVIDIA GPUs, including the NVIDIA’s H100, H200, and B200, and AMD’s MI300X, for concurrency scaling analysis. Using the vLLM framework with the gpt-oss-20b model, we tested how these GPUs handle concurrent requests, from 1 to 512. By measuring…
Time Series Classification Benchmark: Foundation Models vs Classical Methods
We benchmarked 13 time series classification methods, from pretrained time series foundation models to a 22-feature baseline from 2019, on 33 UCR/UEA datasets under one frozen protocol. That is 14,638 recorded method-dataset-resample cells, 11,874 of them scored. The chart compares 12 methods on the 15 univariate datasets every one of them completed, each dataset run…
AI Coding Benchmark: Claude Code vs Cursor
In AI coding, the market has fragmented into two categories: Agentic CLI tools and AI code editors embedded in IDEs. Each claims to automate development. Few comparisons show how they differ under identical workloads. We benchmarked each agent across 10 full-stack web development tasks, performing ~600 atomic validation checks per agent and more than 9,600…
LLM Parameters: GPT-5 High, Medium, Low and Minimal
Some LLMs, such as OpenAI’s GPT-5 family, come in different versions (e.g., GPT-5, GPT-5-mini, and GPT-5-nano) and with various parameter settings, including high, medium, low, and minimal. Below, we explore the differences between these model versions by gathering their benchmark performance and the costs to run the benchmarks. We used the GPT-5 family in our…
HALC-Bench: LLM Hallucination on Long-Context Retrieval Benchmark
HALC-Bench (LLM Hallucination on Long-Context Retrieval Benchmark) measures a large language model’s resistance to fabricating evidence for a metric that does not exist in the target document by using 3 haystacks placed at the beginning, middle, and end of the model’s context window, with 204 questions. claude-fable-5 answered all 204 traps correctly at every haystack…