Agentic AI includes agents that execute complex tasks with minimal human supervision. We evaluated the most popular AI agents, open-source AI agent frameworks, customer service AI agents, and the performance of popular LLMs as AI agents.
We tested leading AI agents across a benchmark that has actual workflow automation needs, including navigating complex interfaces, making precise edits, and completing multi-step processes.
We evaluated four industry leaders on their API keys or playgrounds using a hold-out dataset of 100 questions randomly selected from Bitext Gen AI Chatbot Customer Support Dataset. We created an imaginary company, TechStyle, an e-commerce site with standard policies, and established a small customer database. This info was shared with each AI vendor before we posed our questions.
Our benchmark includes five tasks of increasing difficulty and complexity designed for a human to test success rates with business-specific tasks. The goal of the benchmark is to evaluate document processing by AI agents. We used eighteen different large language models as AI agents.
WebVoyager benchmark evaluates web agents on 15 real-world websites, including Google, GitHub, and Wikipedia. It includes tasks like searching, clicking, navigating, and submitting forms across 643 task instances. Accuracy is measured by successful completion, compared to standard outputs.
