Ekrem Sarı
Ekrem is an AI Researcher and Data Scientist at AIMultiple. He designs and runs hands-on benchmarks for AI and LLM systems.
Professional Experience
At AIMultiple, Ekrem benchmarks end-to-end AI systems and builds the data workflows and dashboards used to track benchmark and product metrics. His benchmarks cover embedding and reranker models, vector and graph databases, inference engines, quantization, GPU concurrency and multi-GPU scaling, cloud GPU pricing and providers, text-to-SQL, and RAG and agentic RAG frameworks.
Before AIMultiple, he worked as a Data Scientist at Yandex, where he queried and analyzed large datasets with SQL to evaluate search and ranking quality against detailed guidelines.
Research Interest
Ekrem's work focuses on measuring how LLM and retrieval systems perform in practice. He designs the test harness, runs the workloads on real hardware, and compares models, frameworks, and infrastructure on accuracy, throughput, latency, cost, and scalability, across the stack from embedding models and vector databases to inference engines and GPU infrastructure. His MSc thesis automates systematic literature reviews with a RAG-based pipeline.
Education
Ekrem holds an MSc in Management Information Systems from Başkent University, where his thesis automated systematic literature reviews with a RAG-based pipeline, and is pursuing a second MSc in Data and Knowledge Engineering at Hacettepe University.
Latest Articles from Ekrem
Disaster Recovery Benchmark: Acronis vs Comet vs MSP360
We benchmarked Acronis Cyber Protect Cloud, Comet Backup, and MSP360 Managed Backup on disaster recovery. Each vendor imaged a live Windows Server 2022 and a live Ubuntu 24.04 server carrying the same deterministic workload, a web service, a 10,000-row database, and 50 files, then recovered the whole machine onto a separate server after a ransomware-style…
Vector Database Benchmark: 7 Open-Source Engines for RAG
We benchmarked seven open-source, self-hosted vector databases as the retrieval layer of a RAG pipeline, each run one at a time on identical bge-m3 embeddings and real medical and technical queries, so the database index was the sole variable. The workload spanned MedRAG-50k, TechQA-28k, and a 2.25M-vector corpus across eight dimensions, from accuracy and retrieval…
Cloud GPU Pricing, Performance & Provider Comparison
Cloud GPU list prices for the same model can differ several times over from one provider to another. We curated the lowest rate, provider, market range, and median for 40+ GPU configurations across all three pricing tiers, plus a throughput-per-dollar benchmark on 10 models. See the most cost-effective GPU for your workload across 13 hyperscaler…
RAG Evaluation Tools: Weights & Biases vs Ragas vs DeepEval
When a RAG pipeline retrieves the wrong context, the LLM confidently generates the wrong answer. Context relevance scorers are the primary defense. We benchmarked five tools across 1,460 questions and 14,600+ scored contexts under identical conditions: same judge model (GPT-4o), default configurations, and no custom prompts. Under standard conditions, WandB, TruLens, and Ragas emerged as…
Embedding Models: OpenAI vs Gemini vs Voyage
We benchmarked 15 English text-embedding models and a BM25 baseline on over 500 manually curated queries across three retrieval domains: legal contracts (CUAD), customer support (IBM TechQA), and healthcare (MedRAG PubMed). Voyage-3.5 ranks first overall. Perplexity Embed V1 0.6b reaches the upper-mid tier at the lowest price point in our benchmark. nDCG@3: Normalized discounted cumulative…
RAG Frameworks: LangChain vs LangGraph vs LlamaIndex
We benchmarked 5 RAG frameworks: LangChain, LangGraph, LlamaIndex, Haystack, and DSPy, by building the same agentic RAG workflow with standardized components: identical models (GPT-4.1-mini), embeddings (BGE-small), retriever (Qdrant), and tools (Tavily web search). This isolates each framework’s true overhead and token efficiency. The benchmark consisted of 100 queries, with each framework running the full set…
Open Source Embedding Models Benchmark for RAG
NVIDIA Llama-Embed-Nemotron-8B leads in accuracy. On cost, Google’s EmbeddingGemma-300m runs roughly 4x cheaper than Nemotron at the cost of a small accuracy loss. nDCG@3: Normalized discounted cumulative gain at cutoff 3. With one relevant document per query, it is 1 / log2(rank + 1) when the gold document lands in the top 3, and 0…
Best RAG Tools, Frameworks, and Libraries
RAG improves LLM responses by grounding them in external data instead of just what the model memorized in training. We benchmarked the components a RAG system is built from and gathered the results in one place, with a practical guide to choosing each part of the stack. See our benchmark results for each RAG component,…
Agentic Search: Benchmark 8 Search APIs for Agents
Agentic search plays a crucial role in bridging the gap between traditional search engines and AI search capabilities. Search APIs are the first layer of an agentic tool, where performance caps the quality of everything downstream. We benchmarked 8 search APIs across 100 real-world AI/LLM queries, evaluating 4,000 retrieved results with an LLM judge that…
Multimodal Embedding Models: Apple vs Meta vs OpenAI
Multimodal embedding models excel at identifying objects but struggle with relationships. Current models struggle to distinguish “phone on a map” from “map on a phone.” We benchmarked 7 leading models across MS-COCO and Winoground to measure this specific limitation. To ensure a fair comparison, we evaluated every model under identical conditions using NVIDIA A40 hardware…
AIMultiple Newsletter
1 free email per week with the latest B2B tech news & expert insights to accelerate your enterprise.