Premium
Services
Premium

Top 10+ Agentic Orchestration Frameworks & Tools

We ran the same five-agent travel-planning workflow on 13 agent frameworks, 100 times each: totaling 1,300 runs. For comparison, we also ran it 100 times in OpenCode, an open-source AI coding agent.

Nazlı Şipi
Nazlı Şipi
updated on Nov 6, 2026
Loading Chart

  • Ten frameworks are close to each other on cost and speed: 1,000 runs cost about 1.4 to 1.7 USD, and a run takes 12 to 14 seconds on average.
  • Google ADK, smolagents and Microsoft Agent Framework stand apart: they cost about 1.6 to 1.9 times as much, and a run takes 1 to 5 seconds longer on average.
  • The cheapest frameworks (LangGraph, LangChain, Agno) took about 1 second longer per run than the fastest group: Strands, OpenAI Agents SDK, AutoGen, Pydantic AI and CAMEL.
Loading Chart

The activity agent had one rule: pick an indoor activity if the day’s rain chance is 50% or higher or rain is forecast; otherwise pick an outdoor activity. All three days were sunny or overcast with a 5 to 10% rain chance, so the correct answer was an outdoor activity for every day.

Findings from agentic orchestration benchmark

When the model thought before answering, it followed the rule every time (388 of 388 runs). When it answered without thinking, it followed the rule in only 38% of runs.

The same pattern holds within each framework. In LangGraph, for example, the model thought in 3 runs and followed the rule in all 3; in the runs where it did not think, compliance was 46%.

The framework changes how often the model thinks. The model thought in 100% of runs in Google ADK and smolagents, 86% in Microsoft Agent Framework and 78% in CrewAI. In the other 9 frameworks the rate was 0 to 9%.

When the model did not think, the format of the input decided compliance: 42 to 53% in frameworks that pass the instruction unchanged, 2% in Strands and 0% in AutoGen.

Why the framework affects thinking

Claude Haiku 5.5 decides on each request whether to think, based on what it receives. Every framework sent the same instruction and the same data, but each framework’s architecture decides what else arrives with them:

  • How much earlier context each agent gets: Pipeline-style frameworks pass the activity agent only the trip details and the forecast (541 tokens, including the instruction). Google ADK and Microsoft Agent Framework pass the whole conversation so far, including earlier tool calls and results (3,579 and 2,439 tokens).
  • Which template wraps the instruction: smolagents puts it inside a long system template with tool-use rules and examples (2,162 tokens). CrewAI turns it into a role, a goal and a “current task” with an expected output.
  • How earlier agents’ outputs are presented: They can arrive as plain input blocks, as chat messages from other agents, or as the agent’s own earlier replies.

Length alone does not explain the result. CrewAI’s input to the activity agent was only 648 tokens, yet the model thought in 78% of runs. AutoGen’s was 1,093 tokens, and the model thought in only 9% of runs. Both the size and the structure of the input appear to matter.

Errors

The model did not ignore the weather: every violation fell on an overcast day, and no run put an indoor activity on the sunny day. The most common violation was the Vatican Museums, which made up 87% of the indoor picks. Without thinking, the model appears to have treated overcast as indoor weather instead of applying the 50% rule. When the model thought, it followed the rule every time.

Frameworks that pass the text unchanged

These frameworks add no text of their own; the agent receives exactly what the developer writes. In our test we wrote the same input for all seven, so the activity agent received an identical request in each. The model thought in 0 to 3% of runs, and compliance was 42 to 54%.

  • LangGraph: Each agent is a node in a state graph. The node reads the earlier outputs from the shared state and builds its message.
  • LangChain: The agents are steps in an LCEL chain. Each step builds its message from the values passed down the chain.
  • Agno: Each agent is a step in a workflow. The step reads the earlier steps’ outputs by name.
  • LlamaIndex: Each agent is a step in an event-driven workflow. Events carry the earlier outputs to the next step.
  • OpenAI Agents SDK: The agents are called one after another in code. The SDK’s handoffs let the model choose the next agent, so we did not use them for a fixed sequence.
  • Pydantic AI: The agents are called one after another in code, following the programmatic hand-off pattern in its documentation.
  • CAMEL: The agents are called one after another in code. Its Workforce feature adds a coordinator agent with extra model calls, so we did not use it.

Frameworks that add their own format

  • CrewAI: Places the instruction in its “role, backstory, goal” template and presents the step as a “current task” with an expected output, followed by the earlier tasks’ outputs. The activity agent’s input was only 648 tokens, yet the model thought in 78% of runs, and 86% of runs followed the rule.
  • smolagents: Every instruction goes inside smolagents’ own long system template, which explains how to call tools, gives examples and rules, and requires the answer through a final_answer tool. With this template, both thinking and rule compliance reached 100%.
  • Google ADK: Earlier agents’ replies, tool calls and tool results arrive one by one as “For context” notes, and each instruction gets an identity line. Given this 3,579-token input, the model thought before every answer and never broke the rule.
  • Microsoft Agent Framework: The agent receives the whole conversation, tool calls included, with earlier agents’ outputs presented as its own previous replies; the option Claude requires adds “please continue.” at the end. Thinking happened in 86 of 100 runs, and compliance reached 92%.
  • AutoGen: In AutoGen’s group conversation, earlier agents’ replies reach each agent as chat messages. The model thought in only 9 runs, and those 9 were the only runs that followed the rule.
  • Strands: Inputs arrive under “Original Task” and “Inputs from previous nodes” headings. In this format the model never thought, and it followed the rule in just 2 of 100 runs.

Some of these formats are configurable; Google ADK and Microsoft Agent Framework, for example, have settings that narrow the conversation each agent receives. We used the default settings in every framework.

These results come from Claude Haiku 5.5, which decides on each request whether to think. Other LLMs may behave differently, so the gap between frameworks can be smaller or larger with another model.

Recommendations from our test

  • Test a rule with the framework and model you plan to deploy. An instruction that works in one framework may not give the same result in another.
  • If a rule matters, use a higher reasoning setting or turn thinking on. At the medium setting we used, the model skipped thinking in most runs; every run where it thought followed the rule, at the cost of more time and tokens.
  • If a rule is simple and exact (for example, “indoor if the rain chance is above 50%”), check it in code instead of leaving it to the model. A small validation step on the agent’s output catches the error in any framework.
  • In frameworks that send each agent the whole conversation, narrowing the context lowers cost. In our test, however, those frameworks also triggered more thinking, so check that rule compliance holds after you narrow it.

Framework overhead and completion time

  • Framework code added 55 to 148 ms per run, about 1% of the total. The rest was the model’s response time.
  • Overhead does not predict speed. Strands, one of the fastest frameworks, had high overhead (115 ms); Microsoft Agent Framework, the slowest, had low overhead (73 ms).
  • Extra model calls matter more. A repeated tool call or an extra step adds one model call, or 1 to 3 seconds.
  • Run time varied by about 2 seconds from run to run within each framework, so differences of less than about half a second between frameworks are within noise.
Get our team to automate one of your business processes with AI agents, free of charge.
Automate a process

What frameworks send to each agent and token use

The travel planner is the last agent in the workflow. It combines the outputs of the four earlier agents into a day-by-day plan and needs only those four outputs, but the frameworks sent it very different amounts of text. The numbers below are prompt tokens per run, summed over all five agents.

Frameworks that pass only the needed inputs

These frameworks run the agents as steps in a pipeline. The planner receives 2 messages: its instruction and the four outputs. Small differences come from how each framework writes the tool definitions.

  • Agno: 6,052 tokens.
  • LangGraph: 6,053 tokens.
  • LangChain: 6,054 tokens.
  • CAMEL: 6,094 tokens.
  • OpenAI Agents SDK: 6,235 tokens. Its tool definitions are longer.
  • Pydantic AI: 6,411 tokens. Its tool definitions are longer.
  • LlamaIndex: 6,590 tokens. Its tool descriptions include the full function signature and docstring.

Frameworks that add a template or extra inputs

  • CrewAI: 7,456 tokens. It wraps each instruction in its “role, backstory, goal” template, frames the step as a task with an expected output, and appends the earlier tasks’ outputs as context.
  • Strands: 7,693 tokens. It repeats the user’s request to every agent as an “Original Task” block. The edges that keep the agents in order also give some agents extra input; the weather agent, for example, receives the flight agent’s output.
  • smolagents: 16,576 tokens. It wraps every instruction in its own large template, which explains tool use and includes examples. The first agent’s prompt was 1,686 tokens, against 173 in most other frameworks.

Frameworks that share the whole conversation

These frameworks run the agents as a team sharing one conversation, so every agent sees what came before it.

  • AutoGen: 8,419 tokens. It passes only the earlier agents’ final replies, as chat messages; the planner received 6 messages.
  • Microsoft Agent Framework: 13,252 tokens. It passes the whole conversation, including earlier tool calls and tool results; the planner received 11 messages.
  • Google ADK: 17,315 tokens. It relays every earlier reply, tool call and tool result as a separate “For context” note; the planner received 10 messages. The weather agent’s input was 6,358 tokens, against about 1,800 in the pipeline frameworks.

Effect on cost and correctness

  • Correctness was not affected. In all 13 frameworks the data reached the planner intact, with one exception in a single Microsoft Agent Framework run.
  • Cost was affected. 1,000 runs cost about 1.4 USD in the pipeline frameworks and 2.4 to 2.8 USD in Microsoft Agent Framework, Google ADK and smolagents.
  • In Google ADK, smolagents, Microsoft Agent Framework and CrewAI, the model thought more often. This made runs longer but raised rule compliance.

Reliability

  • All 1,300 runs completed without errors or timeouts.
  • In 1,299 of 1,300 runs, the data reached the final plan correctly.
  • The only failure was in Microsoft Agent Framework. The travel planner had every flight offer in its input but left the flight fields empty. This may have contributed: Microsoft Agent Framework passes earlier agents’ outputs to the planner as if the planner had written them itself.

Extra model calls

Each extra model call added 1 to 3 seconds to a run.

  • CrewAI: In 12 of 100 runs, the flight agent repeated the same flight search, with the same inputs, after getting the result.
  • smolagents: It accepts an answer only through its final_answer tool. In 14 of 100 runs, the model first replied in plain text, so smolagents needed one more round.

Agentic frameworks vs AI coding agents

To compare the frameworks with an AI coding agent, we also ran the workflow 100 times in OpenCode, an open-source AI coding agent. We used it as a general agent: same model, instructions and tools, with its own coding tools turned off. Each agent ran as a separate OpenCode session, and our code passed each agent’s output to the next one, as in the pipeline frameworks.

OpenCode produced a correct itinerary in all 100 runs and followed the weather rule in every run, as only Google ADK and smolagents did. The difference comes from a few lines OpenCode adds to every agent’s instructions: the model’s name and an environment block with the working folder, the operating system and today’s date.

  • Thinking: These lines grew the activity agent’s input from 541 to 740 tokens, and the model thought before every answer. In the seven frameworks that pass the text unchanged, it thought in 0 to 3% of runs. To rule out a day effect, we sent both versions of the request 30 times on the same day. The plain version never led to thinking; the version with OpenCode’s lines led to thinking in all 30.
  • Cost: 1,000 runs cost $1.70. This is more than the pipeline frameworks ($1.42 to $1.47) and less than Google ADK, smolagents and Microsoft Agent Framework ($2.45 to $2.75). Prompt tokens per run rose to 8,158, against 6,052 to 6,590, because of the added lines and the full tool descriptions that arrive over MCP. The extra thinking also added to the cost.
  • Extra calls: In 7 of 100 runs, the flight agent repeated the same flight search. CrewAI did the same in 12 of 100 runs.
  • Overhead: OpenCode’s own code added 394 ms per run, against 55 to 148 ms for the frameworks.
  • Completion time: A run took 11.47 seconds on average, the lowest of all. 
Don’t miss our benchmarks and data-driven insights. The button opens Google; selecting AIMultiple confirms that you wish to see AIMultiple more often in Google search results.
GoogleAdd as preferred source

Default settings that needed changes

We had to change some default settings to make the frameworks run this workflow:

  • Microsoft Agent Framework: Its sequential chain did not work with Claude at all (HTTP 400). The chain sends each agent a conversation that ends with the previous agent’s reply, which Claude rejects. The ensure_trailing_user_turnoption fixes this by adding a user message at the end of the conversation.
  • Agno: By default, its workflow skips a failed step and still reports the run as successful. We set the step error policy to “fail” so that errors surface.
  • Strands: Its graph starts an agent as soon as any of the agents it depends on finishes. With plain edges, the planner would run too early and several times, so we added conditions to the edges to keep the order.
  • smolagents: It forces the model to call a tool at every step. Claude rejects this when thinking is on, so we set it to “auto”.
  • OpenAI Agents SDK: It uses OpenAI’s Responses API by default. To use it with a Chat Completions endpoint, which most other providers offer, you need its Chat Completions model class.

Pipeline or shared conversation for a fixed workflow

When the order of agents is fixed, each agent needs only a few earlier outputs. In our test, passing just those outputs moved the data between agents as reliably as sharing the whole conversation: 1,299 of 1,300 runs were correct across both groups. The shared conversation in Google ADK and Microsoft Agent Framework more than doubled the prompt tokens per run. Shared conversations fit agents that need to see each other’s reasoning. In a fixed pipeline, passing explicit inputs costs less, and you can see exactly what each agent receives.

Check what your framework sends to the model

Seven of the 13 frameworks sent the activity agent exactly the text we wrote. The other six added their own templates, relayed messages or headings, from about 50 extra tokens in Strands to more than 3,000 in Google ADK. Those additions changed how often the model thought and whether it followed the rule. Before tuning prompts in a multi-agent system, log the final request each agent sends to the model. A logging proxy or the framework’s tracing shows what the framework added.

What sets the speed of a multi-agent workflow

Framework code took about 1% of each run. The rest came from the number of model calls and the time each call took. Three levers follow from this:

  • Fewer model calls: Watch for repeated tool calls and extra steps; each one added 1 to 3 seconds.
  • Smaller inputs: Shorter inputs for each agent reduce tokens and leave the model less to read.
  • Parallel independent agents: In this workflow the flight and weather agents do not depend on each other. Running them in parallel would remove the shorter of the two, roughly 3 seconds. We did not test parallel execution; this estimate comes from the per-agent times.

Benchmark methodology

Every framework ran the same five-agent travel-planning workflow, with the agents always in this order:

  1. Parser: Extracts the origin, destination, departure date and trip length from the user’s request.
  2. Flight finder: Calls a flight search tool and lists the offers by price.
  3. Weather reporter: Calls a weather tool for the trip dates.
  4. Activity recommender: Picks one activity per day, using the indoor/outdoor rule above.
  5. Travel planner: Combines the four outputs into a day-by-day plan with the cheapest flight.

The user request was: “I want to travel from Berlin to Rome on October 27, 2026. I will stay for 3 days.” Each agent had the same written instruction in every framework and returned its answer as JSON. Only the flight finder and the weather reporter had tools.

Tool data

Both tools returned fixed data, so every run in every framework received identical tool results:

  • Flights: Six one-way Berlin to Rome offers. These are synthetic: the plausible carriers, times and prices do not come from a live fare search.
  • Weather: A real Rome forecast for October 27 to 29, 2026, recorded once from WeatherAPI.com.

Because the tools answered instantly, completion times reflect model calls and framework code, not API latency.

Frameworks

  • Versions tested: LangGraph 1.2.17, LangChain 1.4.7, CrewAI 1.15.30, AutoGen 0.7.5, Microsoft Agent Framework 1.23.0, OpenAI Agents SDK 0.24.0, Google ADK 2.13.0, Agno 3.1.10, Pydantic AI 2.61.0, smolagents 1.26.0, LlamaIndex 0.14.25, Strands Agents 1.62.0 and CAMEL 0.2.90, all on Python 3.12.
  • How the agents were run: Each framework ran the agents with its own multi-agent construct where it has one. OpenAI Agents SDK, Pydantic AI, smolagents and CAMEL have no construct for a fixed sequence, so their agents were called one after another in code.
  • Settings: All settings stayed at their defaults, except the changes listed under “Default settings that needed changes”.

Model and runs

  • Model: Every agent in every framework used Claude Haiku 5.5 through OpenRouter, served by Anthropic, with the reasoning setting fixed at medium. Settings a framework tried to add, such as temperature, were removed before the request reached the model.
  • Runs: Each framework ran the workflow 100 times, 1,300 runs in total, on October 27, 2026. Runs were executed one at a time, in a shuffled order across frameworks.
  • Warm-up: Before each timed run, the same workflow ran once against a scripted test model, so that one-time loading costs did not count toward the measured time.
  • Limits: Each run was capped at 30 model calls and 300 seconds. No run reached either limit.

What we measured

  • Requests: A logging proxy recorded every request and response between the frameworks and the model. Each agent had its own endpoint, so every model call could be traced to the agent that made it. Tool calls were logged separately.
  • Completion time: Time from the start of the workflow to the final plan.
  • Tokens and cost: Prompt, completion and thinking tokens and the cost of each call, as reported by the provider.
  • Framework overhead: The part of a run spent neither waiting for the model nor running tools.
  • Thinking: A run counts as one where the model thought if the provider reported thinking tokens for the activity agent’s call.
  • Correct itinerary: The final plan contains the cheapest flight (flight number and price), the three trip dates, each day’s weather as forecast, and each day’s activity and setting exactly as the activity recommender produced them.
  • Rule compliance: All three days follow the indoor/outdoor rule. Since no day had a rain chance near 50%, the correct answer was an outdoor activity for every day.

OpenCode: OpenCode 1.19.5 ran as a local server. Each agent was a separate session with its own instruction, and our code passed the outputs from one agent to the next. The tools reached it over MCP. Its own tools (shell, file editing, web search and others) were turned off, and the prompt-caching markers it adds were removed before requests reached the model.

Cite this benchmark

Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.

Nazlı Şipi (2026) - "Top 10+ Agentic Orchestration Frameworks & Tools". Published online at AIMultiple.com. Retrieved November 6, 2026, from: https://aimultiple.com/agentic-orchestration [Online Resource]

Şipi, N. (2026, November 6). Top 10+ Agentic Orchestration Frameworks & Tools. AIMultiple. https://aimultiple.com/agentic-orchestration

@misc{sipi2026,
  author = {Şipi, Nazlı},
  title  = {{Top 10+ Agentic Orchestration Frameworks & Tools}},
  year   = {2026},
  month  = nov,
  howpublished    = {\url{https://aimultiple.com/agentic-orchestration}},
  note   = {AIMultiple. Retrieved November 6, 2026}
}
Download all data

Results and timestamps of 11.3 thousand data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 4 CSV files and a README.

Last updated: November 10, 2026
Download

Want the granular data behind it? Join Premium

Changelog

4 updates
  1. Removed the "Agentic orchestration benchmark" section.

  2. Updated the 'Agentic orchestration benchmark' section with new metrics.

Nazlı Şipi
Nazlı Şipi
AI Researcher
Nazlı is a data analyst at AIMultiple. She has prior experience in data analysis across various industries, where she worked on transforming complex datasets into actionable insights.
View Full Profile

Be the first to comment

Your email address will not be published. All fields are required. Comments are left in their original language.

0/450