AI Agents
AI agents are software systems that use reasoning, planning, and tools to assist or automate complex tasks. We compare the top open-source and commercial agents.
- AI AgentsBenchmarkOct 9
AIM-Code Bench: Agentic Coding Benchmark
Evaluating LLMs without their coding-agent harnesses does not fully reflect how they are used in practice. Real development also involves extending existing work, taking over unfamiliar code, and leaving implementations that others can build on. We introduce AIM-Code Bench to evaluate models together with their coding-agent harnesses across evolving software tasks. Beyond correctness, the benchmark…
- Agentic WebBenchmarkOct 9
Video Search API Benchmark: Bright Data vs YouTube
We compared Bright Data Video Search with YouTube search on 50 English queries across five types of queries. Both engines were queried to exhaustion, and a three-model panel assessed sampled video frames. Usable clips per search estimates how many returned videos contain the queried subject or action. It combines the number of unique videos returned…
- AI AgentsInsightSep 29
AI Agent Traps: 20 Real-Life Incidents
AI agent adoption has outpaced AI agent security: 82% of enterprises now deploy agents, but 44% have policies to secure them, and one in five organizations has already experienced an agent-related breach. We analyzed 20 real-world security incidents and found that behavioral control and systemic traps (not prompt injection) now drive the majority of critical…
- Agentic FinanceBenchmarkSep 28
AI-Based Stock Trading: Which Gen AI Tool Is Better
LLM tools have been used in AI-based stock trading since their emergence. I tested 20 generative AI models on a single, hard forecasting task: predict the 3-day cumulative abnormal return (CAR) of 132 stocks after an unexpected event, using only the firm-level information supplied in the prompt. For further details on the benchmark, read the…
- AI AgentsBenchmarkSep 21
AI Agent Performance: Success Rates & ROI
Recent research reveals that AI performance follows predictable exponential decay patterns, enabling businesses to forecast capabilities and differentiate between costly failures and successful ROI-generating implementations. I oversaw 12 AIMultiple benchmarks, including nearly 70 AI agents across more than 1,000 tasks. See what each benchmark measures and where limits remain: Computer use agents interact with a…
- AI AgentsSep 21
Agent benchmarks
Agent benchmark
- AI AgentsBenchmarkSep 21
Mobile AI Agents Tested Across 65 Real-World Tasks
We spent 3 days benchmarking four mobile AI agents (DroidRun, Mobile-Agent, AutoDroid, and AppAgent) across 65 real-world tasks using an Android emulator with applications such as calendar management, contact creation, photo capture, audio recording, and file operations. See benchmark results including real-world performance comparison, costs and execution times: Highest success rate (43%) with high cost…
- Agentic WebBenchmarkSep 17
Top 4 AI Search Engines Compared
Searching with LLMs has become a major alternative to Google search. We benchmarked the following AI search engines to see which one provides the most correct results: Deepseek is the leader of this benchmark, by correctly providing 57% of the data in our ground truth dataset. You can also read our AI deep research benchmark…