Top 13 Open-Source Agentic AI Frameworks

smolagents is not in the chart. At 20.55 s and $4.52 per 1,000 runs on average, it had by far the highest time and cost, and including it made the differences between the other frameworks unreadable. We explain why below.

• The three cheapest frameworks are Agno ($1.41), CAMEL ($1.45) and Google ADK ($1.48). All three run tools one after another and never hit the errors that come from tools not waiting for each other. LangChain ($1.49) and LangGraph ($1.50) run tools at the same time but cost about the same, because they write some of the shortest tool definitions.
• Microsoft Agent Framework is fast (7.57 s) but the most expensive framework in the chart ($1.88). It runs tools at the same time, so 64% of its runs hit an error and repeat steps, and it writes long tool definitions.
• The cost gap between the 12 frameworks in the chart is small: at most $0.47 per 1,000 runs.
• 98% to 99% of the time goes to waiting for the model. The framework’s own code has almost no effect on time.

We first ran this benchmark in February 2026 with four frameworks on GPT-5.2: LangChain, LangGraph, AutoGen and CrewAI. CrewAI then took about twice as long as the other three (52.7 s on average against 27.5-29.6 s), and the heaviest framework used 64% more tokens than the lightest. With current versions, the four frameworks average 8.20-8.39 s and the token gap is 26%.

See our benchmark methodology for details on how we ran the tests and analyzed the results.

Parallel and sequential tool execution

The model sometimes asks for several tools at once. Agno, CAMEL and Google ADK run these tools one after another: the next one starts when the previous one finishes. The other ten frameworks start them all at the same time by default. When the tools depend on each other, starting them together causes problems, because the second tool does not wait for the first one to finish. In a real application, this is like one tool writing a database record while another tool tries to read the same record.

The sequential behavior applies to the synchronous calls we used. Agno’s arun(), CAMEL’s astep() and async tools in Google ADK run the calls at the same time.

We saw this on three tasks:

• Task 1 (load the data, then calculate):
  • The model asked for both steps together. In the frameworks that run tools at the same time, the calculation started before the data was loaded and failed, and the model asked for it again.
  • That is one extra LLM call. The task took 2.5-3 s and $0.48-0.64 per 1,000 runs in Agno, Google ADK and CAMEL, against 3.5-5.2 s and $0.74-1.05 in the ten frameworks that run tools at the same time.
• Task 3 (apply two filters one after the other, then calculate):
  • The model asked for all three steps together. The calculation ran before the second filter finished, so it used 1,483 customers instead of 592.
  • 10 of the 13 failed runs in the benchmark come from this: 3 wrong answers each in LangChain and LangGraph, 1 in Microsoft Agent Framework, and 3 CrewAI runs stopped by a tool error. In 6 of the wrong answers, the model said the result was unreliable instead of giving a number.
  • The model often noticed the inconsistency and repeated steps. Microsoft Agent Framework repeated steps in 59% of runs and was the most expensive framework on this task after smolagents ($1.96 per 1,000 runs). Google ADK never repeated a step and was the fastest (8.05 s).
• Task 5 (recognize the JSON and text columns and process each with the right tool):
  • Here the frameworks did not differ. In all 1,300 runs, the model inspected both columns before processing them and chose the right tool.
  • The difference again came from parallel execution. In 85% of CrewAI runs and 69% of Microsoft Agent Framework runs, a tool ran before its data was ready and steps were repeated.
  • The answers were correct, but CrewAI was the most expensive framework on this task after smolagents ($2.08 per 1,000 runs).

We used each framework’s default setting. Running tools at the same time saves time only when tools are slow, such as API calls that take seconds each. Our tools finish in milliseconds, so here it brought the errors above and no speed gain. If your tools are slow and independent, keep the default. If they share data, there are two options:

• Configure the framework to run tools one at a time. SequentialToolExecutor in Strands Agents and max_tool_threads=1 in smolagents do this. In LlamaIndex, allow_parallel_tool_calls=False makes the model request one tool per response.
• Write tools that do not depend on each other, for example by combining “filter” and “calculate” into one tool.

How instructions and history reach the model

Most frameworks send our instructions as they are and send earlier steps as standard tool messages. smolagents places our instructions inside its own 6,780-character template and sends earlier steps again as plain text. This difference showed up in three places:

• The first request is larger: 4,428 tokens on average in smolagents, against 1,917-2,822 in the other frameworks.
• The model thought more: the same model produced 1,050 reasoning tokens per run on average in smolagents, against 122-228 in the other frameworks. Reasoning tokens are billed as output and take time to generate. In our agentic orchestration benchmark, the model also thought in every smolagents run.
• There are more LLM calls: 6.52 per run on average in smolagents, against 3.73-4.78 in the others. Most of this comes from its different behavior on Task 4 (below).

The gap grows on tasks with more steps. Task 2 asked the agent to calculate and compare two groups: the revenue lost from customers with and without tech support. The agent filters and calculates each group separately, so the task takes several steps, and the template and history are sent again at every step.

• smolagents sent 31,200 tokens per run on average on this task; the other frameworks sent between 9,244 and 14,161.
• smolagents finished the task in 24.34 s and at $4.38 per 1,000 runs, 2-3 times the other frameworks. Two smolagents runs hit the LLM call limit without an answer.
• Most of the other frameworks finished in 10-12 s. Google ADK was the fastest at 7.58 s with 3.39 LLM calls, and CAMEL was the cheapest at $1.34.

We used smolagents’ default template. You can pass a shorter one with the prompt_templates parameter.

Behavior when a tool fails

Task 4 tested what the agent does when a tool is broken. The tool that calculates churn rate by payment method fails on the first three calls (connection error, timeout, rate limit) and works on the fourth. Every error message says “retry”. The agent has two options: retry the tool, or filter and calculate each payment method separately.

• In twelve frameworks, the model retried the tool after each error in 97% to 100% of runs and got the result on the fourth try. The task took 5.7-7.1 LLM calls and 7.9-10.4 s on average.
• In smolagents, the same model gave up on the tool after the second or third error in 96% of runs and calculated the four payment methods one by one. The task took 13.76 LLM calls and 41.07 s on average, and cost $9.96 per 1,000 runs, against $1.90-2.75 in the other frameworks.
• The error message was the same in every framework. What differs in smolagents is that it shows the error and earlier steps as plain text inside its own template. We think the behavior difference comes from this, but we did not test it separately.

The model also affects this behavior: in a check with GPT-5.2 and one run per framework, 6 of the 13 frameworks gave up on the tool and took the other route.

We showed errors to the model as they were. If temporary errors such as connection errors, timeouts and rate limits are retried inside the tool, the model never sees them, and the behavior no longer depends on the framework.

How tool definitions are written for the model

We gave every framework the same Python functions with the same descriptions. Each framework turns them into the tool definitions the model reads, in its own way. This changed the size of the first request: 1,917 tokens on average in Agno, 2,754 in AutoGen. Most of the difference comes from the tool definitions:

• Agno, LangChain and LangGraph use only the first sentence of the function description as the tool description and put the parameter descriptions on the parameters. Their definitions are the shortest.
• AutoGen, Microsoft Agent Framework, CrewAI and Google ADK send the full function description, including the parameter and return descriptions.
• LlamaIndex also adds the function signature at the start.
• Pydantic AI writes the description in XML tags and adds the return description.

Two frameworks add text to the instructions. CrewAI places the instructions in a role, backstory and goal template, and Google ADK adds a line that starts with “You are an agent” at the end. These additions are smaller than the differences in the tool definitions.

Tool definitions are sent again with every request, so the difference adds cost at every step. Agno and AutoGen made almost the same number of LLM calls per run (4.40 and 4.45), yet AutoGen sent 37% more tokens to the model on average (14,932 against 10,899).

We gave every framework the same functions and descriptions. Keeping function descriptions short, or giving the framework a separate tool description, reduces this difference. In CrewAI, the text added to the instructions can be changed with system_template and prompt_template.

Framework code overhead

The framework’s own code adds 0.04-0.14 s to a run on average, less than 2% of the total time.

Startup takes about 1 s in Google ADK, 0.6 s in CAMEL and under 0.35 s in the others. Startup time is not included in the benchmark results.

In systems that start fresh for every request, such as serverless functions, startup time is added to every request. Loading the framework once when the application starts and keeping the process running removes it.

Multi-agent support

This benchmark used a single agent, but all 13 frameworks support running several agents together. We checked the information in this section and the next three against the source code of the versions we tested. The approaches fall into three groups:

• Defined as a flow: LangGraph (nodes and edges), Microsoft Agent Framework (WorkflowBuilder), Google ADK (SequentialAgent, ParallelAgent, LoopAgent), Strands Agents (Graph and Swarm) and LlamaIndex (AgentWorkflow with handoffs between agents).
• Defined as a team: CrewAI (a sequential Crew or one with a manager agent), AutoGen (RoundRobinGroupChat, SelectorGroupChat, Swarm), Agno (Team) and CAMEL (Workforce, RolePlaying).
• Handing work to another agent: OpenAI Agents SDK (handoffs), smolagents (managed_agents), Pydantic AI and LangChain (giving one agent to another as a tool).

We measured how these frameworks handle a five-agent flow in our agentic orchestration benchmark.

MCP support

MCP (Model Context Protocol) gives agents a standard way to connect to external tools and data sources. All 13 frameworks can use tools from MCP servers.

MCP client included in the framework’s own package:

• Agno
• CAMEL
• CrewAI
• Google ADK
• LangChain (extra dependency: langchain[mcp])
• Microsoft Agent Framework
• OpenAI Agents SDK
• Pydantic AI
• smolagents (extra dependency: smolagents[mcp])
• Strands Agents

Separate package needed:

• AutoGen: autogen-ext
• LangGraph: no MCP client of its own; uses LangChain’s MCP support or langchain-mcp-adapters
• LlamaIndex: llama-index-tools-mcp

Don’t miss our benchmarks and data-driven insights. The button opens Google; selecting AIMultiple confirms that you wish to see AIMultiple more often in Google search results.

Add as preferred source

Human approval (human-in-the-loop)

Some actions should not happen without a person’s approval, such as sending an email or deleting a record. The frameworks support this in three ways.

Asking for approval before a tool runs:

• Agno: requires_confirmation
• Google ADK: require_confirmation
• LangChain: HumanInTheLoopMiddleware
• Microsoft Agent Framework: approval_mode
• OpenAI Agents SDK: needs_approval
• Pydantic AI: requires_approval

Pausing the flow and waiting for a person’s answer:

• LangGraph: interrupt
• LlamaIndex: events that wait for human input
• Strands Agents: interrupts

Connecting a person as an agent or a tool:

• AutoGen: UserProxyAgent
• CAMEL: HumanToolkit
• CrewAI: feedback through human_input at the end of a task
• smolagents: UserInputTool

Memory

Every framework keeps the history within a task. The difference is in persistent memory across tasks and conversations.

Built-in persistent memory:

• Agno: MemoryManager
• AutoGen: Memory
• CAMEL: LongtermAgentMemory
• CrewAI: the memory setting on a Crew
• Google ADK: memory services
• LangChain: checkpointers and stores
• LangGraph: checkpointers and stores
• LlamaIndex: Memory
• Microsoft Agent Framework: context and history providers
• OpenAI Agents SDK: sessions
• Strands Agents: session managers

No built-in persistent memory (the application stores the history and passes it to the agent on the next run):

• Pydantic AI
• smolagents

Best agentic frameworks

LangGraph

It defines an agent as a graph of nodes and edges, and the developer manages the state passed between steps explicitly. It has Python and JavaScript versions, and runs can be traced in LangChain’s LangSmith platform.

The longest LangGraph run on Task 3 made 25 tool calls before reaching the correct answer, and 16 of them were repeats. Apart from smolagents, the longest run in any other framework stopped at 21 tool calls.

LangChain

The LangChain team’s general-purpose framework for LLM applications. Since version 1.0, agents are built with create_agent and run on the LangGraph runtime. Middleware can change the agent loop, and ready-made middleware covers summarizing long histories, masking personal data, retrying after errors and falling back to another model.

Because create_agent runs on LangGraph, it sent the model the same instructions and tool definitions as LangGraph, and the two frameworks ended up with almost the same time and cost.

CrewAI

It defines agents by role, goal and backstory and runs them with tasks as a team (a crew). On top of crews, Flows connect steps through events, and the company also offers a paid enterprise platform.

It was the fastest framework on Task 4 (7.85 s) and made the fewest LLM calls there (5.68 on average).

AutoGen

Microsoft Research’s framework in which agents work by sending messages to each other. It has a layered design (Core, AgentChat and Extensions), and AutoGen Studio, a no-code interface, comes as a separate package. Microsoft recommends Microsoft Agent Framework for new projects.

By default, AssistantAgent stops after one round of tool calls and returns the tool result as its answer (max_tool_iterations=1). Multi-step tasks need a higher value, and we raised it.

Microsoft Agent Framework

Microsoft’s framework that brings AutoGen and Semantic Kernel together. It supports Python and .NET. It offers OpenTelemetry tracing and middleware for agent calls, and a development interface called DevUI comes as a separate package.

It was the fastest framework on Task 5 (6.50 s), even though it often repeated steps.

OpenAI Agents SDK

OpenAI’s framework and the successor to Swarm, built around agents, tools, handoffs and guardrails. It has Python and TypeScript versions. By default, it sends a trace of every run to OpenAI’s tracing dashboard (we turned this off for the benchmark). It includes separate modules for voice and realtime agents. It uses OpenAI’s Responses API by default, so models from other providers need the Chat Completions model class.

Google ADK

It works with Gemini directly and with other models through LiteLLM, which is what we used. It comes with a local development interface (adk web) and an evaluation tool (adk eval), and agents can be deployed from the command line to Vertex AI Agent Engine, Cloud Run or GKE.

It was the fastest framework on average (6.79 s) and placed among the three fastest on all five tasks. It also made the fewest LLM calls per run (3.73 on average).

Agno

It ships with a large catalog of ready-made toolkits and offers AgentOS, a runtime for serving and managing agents as an API on your own infrastructure.

In Agno, the model almost never wrote commentary next to its tool calls: it did so in 0.2% of tool-calling turns, against 3% to 26% in the other frameworks. This commentary is billed as output tokens.

Pydantic AI

It validates tool arguments and agent output with Pydantic models and asks the model to correct an invalid argument. A typed deps mechanism passes dependencies such as database connections to tools. Runs can be traced with Pydantic Logfire and made durable against interruptions with Temporal, DBOS or Prefect.

smolagents

Its best-known approach is CodeAgent, which calls tools by writing Python code instead of JSON and runs that code locally or in isolated environments such as E2B, Docker or Modal. Tools and agents can be shared on the Hugging Face Hub. We tested ToolCallingAgent, which makes JSON tool calls, so CodeAgent is not part of these results.

Because it returns the answer through a separate final_answer tool, 99.6% of its outputs were plain JSON. In the other frameworks, 17% to 41% of answers came wrapped in a markdown code block.

LlamaIndex

It includes components for loading, splitting, indexing and querying documents, and a query engine can be given to an agent as a tool with QueryEngineTool. For agents, it offers FunctionAgent and AgentWorkflow, which connects several agents. It has Python and TypeScript versions.

It was the most expensive framework in the chart on Task 2 and Task 4, the tasks with the most steps ($1.91 and $2.75 per 1,000 runs). Its long tool definitions are sent again at every step, so the gap grows with the number of steps.

Strands Agents

It leaves planning and tool choice to the model and keeps the agent loop simple. If no model is set, it uses a Claude model on Amazon Bedrock, and it has built-in providers for Anthropic, OpenAI, Gemini, Mistral, Ollama and LiteLLM. It traces runs with OpenTelemetry.

In all 100 Task 1 runs, the model first called the calculation tool without loading the data, got an error, and loaded the data in a separate step. Because of this extra step, Strands was the slowest framework in the chart on Task 1 (5.18 s).

CAMEL

An agentic framework from the CAMEL-AI community that grew out of multi-agent research; OWL is built on it. Besides ready-made toolkits, it includes modules for synthetic data generation and benchmarks, so it also targets research use.

It was the slowest framework in the chart on Task 3 (14.43 s). On this task, the model made 4.73 LLM calls and produced 598 reasoning tokens per run on average in CAMEL, against 3.22 and 207 in Google ADK.

Benchmark methodology

All tasks use the same data: 5,000 customers sampled from the IBM Telco customer churn dataset. We added a JSON column (device data) and a text column (support notes) to the dataset. The agent first loads the data, then filters and calculates with tools, and returns the answer in a fixed JSON format.

• Task 1, basic calculation: Churn rate by contract type. Measures how much overhead the framework adds to the simplest job.
• Task 2, comparing two groups: Monthly revenue lost from customers with and without tech support. Measures whether two separate results stay correct across steps.
• Task 3, thresholds: Churn rate of customers with under 1 year of tenure who pay over $70 a month. Measures turning phrases into the right numbers and applying two filters one after the other.
• Task 4, broken tool: Churn rate by payment method. The required tool fails three times and works on the fourth. Measures what the agent does when a tool fails.
• Task 5, mixed data: Device counts (from the JSON column) and complaint keywords (from the text column) for churned customers who pay over $100 a month. Measures whether the agent recognizes the column types and picks the right tool.

Setup

• Model: Claude Haiku 5.5 through OpenRouter with the Anthropic provider, reasoning effort medium.
• 100 runs per framework and task, 6,500 runs in total. Date: October 27, 2026.
• Limits per run: 15 LLM calls on Tasks 1-3, 25 on Tasks 4-5, and 420 s.
• Every framework received the same instructions, the same task text and the same tools. We set up each framework the standard way and did not change its default behavior. We only raised step limits enough for the tasks to finish, and changed one setting each in CrewAI, smolagents and AutoGen so they could work with Claude.
• All requests went through the same logging proxy. The proxy applied the same model and reasoning setting to every request and removed sampling settings that some frameworks add, such as the temperature of 0.1 that LlamaIndex sends.
• Each run ran in its own process after an unmeasured warm-up pass. Runs ran three at a time in shuffled order.
• We compared each answer with the correct answer calculated by the same tools.
• The results for our 5,000-customer sample differ from the dataset’s well-known public results, so the model cannot answer from memory.
• Versions tested: LangGraph 1.2.17, LangChain 1.4.7, CrewAI 1.15.30, AutoGen AgentChat 0.7.5, Microsoft Agent Framework 1.23.0, OpenAI Agents SDK 0.24.0, Google ADK 2.13.0, Agno 3.1.10, Pydantic AI 2.61.0, smolagents 1.26.0, LlamaIndex 0.14.25, Strands Agents 1.62.0, CAMEL 0.2.90.

Cite this benchmark

Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.

@misc{dilmegani2026,
  author = {Dilmegani, Cem and Şipi, Nazlı},
  title  = {{Top 13 Open-Source Agentic AI Frameworks}},
  year   = {2026},
  month  = oct,
  howpublished    = {\url{https://aimultiple.com/agentic-frameworks}},
  note   = {AIMultiple. Retrieved October 30, 2026}
}

Changelog

10 updates

All10General9Methodology1

20262025

1. Mar 2

   Replaced the benchmark results and analysis in the "Agentic AI frameworks benchmark" section.

2. Jan 22

   Removed a methodology change that described how tasks were integrated into agent-based architectures.

Cem Dilmegani

Principal Analyst

Follow On

Cem has been the principal analyst at AIMultiple since 2017.

Cem's work at AIMultiple has been cited by leading global publications including Business Insider, Forbes, Morning Brew, and Washington Post, global firms like Deloitte and HPE, NGOs like World Economic Forum, and supranational organizations like European Commission. [1], [2], [3], [4], [5]

Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.

He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.

Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.

View Full Profile

Researched by

Nazlı Şipi

AI Researcher

Follow On

Nazlı is a data analyst at AIMultiple. She has prior experience in data analysis across various industries, where she worked on transforming complex datasets into actionable insights.

View Full Profile

Comments 1

Share Your Thoughts

Your email address will not be published. All fields are required. Comments are left in their original language.

Name

Email Address

Comment

0/450

Post Comment

Chaitanya

Dec 19, 2025 at 01:47

Thank you for this informative and detailed article! It helped me get a reading on these frameworks.

Reply