Agentic Data Benchmark: Firecrawl, Apify & Zyla
We ran 200 tasks through 4 data layers for IA agents and Claude’s built-in web search: 100 on open data sources and 100 on protected domains, in 2.600 agent runs and 10.811 API calls. We wanted to see what a data layer changes compared with the model’s own search, and how the data layers differ from each other.
Firecrawl ran twice, with and without Alexandria, to measure the catalog’s effect within the same product.
Read our agentic data benchmark methodology to learn more about our testing process.
Agentic data benchmark results
We connected each product to the same agent, Claude Haiku 5.5 at high reasoning effort, through the product’s MCP server, and compared it with Claude’s built-in web search. With a product connected, the agent had no web tool of its own, so every answer came through that product.
- Both Firecrawl setups cost less than Claude’s web search in both question sets, LLM cost included: $0,73 to $1,36 per 100 questions against $2,93 to $3.37.
- Claude’s web search had the lowest accuracy on protected domains (49%) and the second lowest on open data sources (53%), above only Zyla.
- On open data sources, Firecrawl with Alexandria scored 0.7 points above Firecrawl web-only (99,7% vs. 99,0%), within the margin of error. On protected domains the gap widened to 5.3 points (85,3% vs. 80,0%), mostly on YouTube questions.
- Apify answered 90,0% of open data questions correctly, but 100 questions cost $8,75, more than ten times Firecrawl’s cost. Actor charges made up $8,33 of that. On protected domains, Apify scored highest, 98,0%, for $1,11 per 100 questions, less than Alexandria’s $1.36. Apify Store has an Actor built for each of the four platforms, and the agent needed 1.25 Actor runs per question.
- Zyla answered 43,7% of open data questions correctly for $5.92. Our search of its catalog found no API for the PyPI and Crossref questions, and it got none of those 18 questions right. On protected domains it rose to 80,3% for $5,57, the highest cost of the four data layers. It called an API in every run, and Facebook was its weakest platform (52%): its Facebook API returns the rounded count the page displays, such as 10M for 10.838.386.
We also asked 22 trap questions that have no answer: 10 on open data sources, such as a CVE ID that does not exist or an unemployment rate for a month BLS never published, and 12 on protected domains about accounts that do not exist.
In 286 runs, one answer made up a value: in one of three runs, Firecrawl with Alexandria gave 0 followers for a nonexistent X account, read from a third-party site that mirrors X profiles.
Sources of correct answers
Alexandria adds source-specific connectors to Firecrawl’s search and scrape tools: App Store, GitHub and SEC filings for open data, and YouTube, X and Influencers Club for social media accounts. Apify Store and Zyla API Hub are marketplaces of ready-made scrapers and third-party APIs, and both list scrapers or APIs for Instagram, X, Facebook and YouTube. They have no separate search tool, but both catalogs list web search Actors or APIs.
- On open data sources, Alexandria used its catalog in 15% of runs: 23% on count-or-compare questions and 8% on one-value questions. All 41 of those answers were correct.
- On protected domains, which restrict automated access, the share rose to 46%, about half of the runs. Alexandria called the catalog in 61 of 66 YouTube runs but in 2 of 66 Instagram runs.
- Against Firecrawl web-only on the same questions, catalog runs cost 34% more and were 8 seconds faster on open data sources. On protected domains, they cost 2.5 times as much and took about the same time. On count-or-compare questions, they were faster in both sets: 43 vs. 59 seconds and 40 vs. 53 seconds.
- Catalog runs were also more accurate than web-only on the same questions: 100% vs. 95,1% on open data sources and 86,1% vs. 72,1% on protected domains. On protected domains, 26 of Alexandria’s 43 wrong answers came from runs with no catalog call, 20 of them on Instagram.
- Apify and Zyla did not search the web through their catalogs, apart from 2 Google search Actor calls by Apify on protected domains. Apify’s correct answers all came from single-source Actors in both sets. Actor fees were 95% of its cost on open data sources and 78% on protected domains.
- Zyla called no API at all in 92 of its 169 failed runs on open data sources. On protected domains, it called an API in every run.
Accuracy and cost by question type
Question examples:
One record, one value: Which license does the GitHub repository tmux/tmux use? Give its SPDX identifier.
One record, two values: What is the tag of the latest release of the GitHub repository asmvik/yabai, and on what date was it published?
Many records, count or compare: In the US App Store, which of these three iOS apps has the most user ratings: Costco, SkyView or YouTube? On what date was that app’s current version released?
- Firecrawl Alexandria and Firecrawl web-only scored 95,8% on count-or-compare questions but 68,8% and 55,2% on two-value questions. On Instagram’s two-value questions, they scored 12% and 21%: both gave the 12 posts visible on the profile page as the account’s post count.
- On YouTube’s two-value questions, Alexandria scored 100% and web-only 62%. Alexandria used its catalog in 61 of 66 YouTube runs.
- On protected domains, Apify answered 93,8% of one-value questions and all two-value and count-or-compare questions correctly, at $0,74 to $1,83 per 100 questions. On open data sources, its accuracy fell and its cost rose as questions got harder.
- Zyla scored 82,3% and 81,2% on one-record questions and 66,7% on count-or-compare questions. Count-or-compare questions cost it $9,20 per 100 questions, the highest of any product on this question set.
- Claude web search scored 50,0% and 25,0% on one-record questions and 54,2% on count-or-compare questions.
Time and tokens per question
- On protected domains, Alexandria used 106.000 tokens per question against 62.000 for web-only, and took 23.5 seconds against 21.8. Its catalog calls returned more text than web calls: 12.000 characters against 7.700.
- Apify was again the slowest data layer, at 24.1 seconds, but only 0.6 seconds behind Alexandria. 16.3 seconds of that went to Actor runs. It used 52.000 tokens, half of Alexandria’s, over 4.5 model calls per question.
- Zyla was the fastest data layer at 18.2 seconds. It made 5.7 model calls per question and used 46.000 tokens, the fewest of the four.
- Claude web search took 7.7 seconds and used 29.000 tokens. It again had the highest LLM cost per question, $0,028, from 2.5 searches per question billed at about 1 cent each.
- Output stayed at 1-3% of the tokens for every product in both sets.
Choosing a data layer by workflow
Which product fits depends on the sources your workflow reads:
- For open data from official sources, a web search and page reader was enough, and a catalog mostly added cost. In Firecrawl Alexandria, a call to a gratuito public source such as GitHub or the SEC costs 5 credits, a page scrape 1 credit, and Similarweb’s traffic data 110 credits.
- For social media accounts, a scraper built for the platform did better than reading the page. A profile page shows only the posts loaded in its grid and rounds large counts, while Apify’s platform Actors return the exact numbers as JSON fields.
- Alexandria’s catalog helped where it had a provider for the platform, such as YouTube. On Instagram, the agent used the catalog in 2 of 66 runs and read the page instead.
- A marketplace covers only the sources its sellers list. Apify Store had an Actor for all 10 of our open data sources and all 4 platforms. Zyla had no API for Crossref or PyPI, and its Facebook API returns the rounded count the page displays.
- If a workflow touches a few known sites, a dedicated scraper with a fixed output fits better. If the sources are not known in advance, a web search tool with a catalog behind it reaches more of them.
How each data layer works
The three products give the agent different things to call:
Firecrawl
Firecrawl’s MCP server pairs its web search and scraping with Alexandria, a catalog of 570+ data providers in 35 categories.
- Every web search also returns the Alexandria providers that match the query. A page scrape returns the page alone, so an agent that starts from a known URL sees the catalog only if it searches or looks the catalog up.
- Sources include official data owners, registries, licensed publishers and Firecrawl’s own indexes, as well as social media data vendors such as Influencers Club and ScrapeCreators. Outside developers cannot add their own tools, unlike on Zyla and Apify.
- Licensed sources, such as Benzinga and Apollo, are called through the same Firecrawl account. Some of them need an organization admin to accept their terms before the first call.
- When the catalog has nothing that fits, Firecrawl’s own guidance is to scrape the live web with the same server.
Apify
Apify’s MCP server opens Apify Store, where Apify and independent developers publish Actors, and it tells the agent how to use them.
- The server tells the agent to search the Store first, read an Actor’s input format before running it, and prefer Actors with more users and higher ratings.
- One site can have dozens of Actors, some built by Apify and some by the community, each with its own inputs and output fields. For platforms such as Instagram and Facebook, Apify maintains its own Actors, which return profile fields such as exact follower counts as JSON.
- Running an Actor returns a run status and a dataset ID, and the agent reads the results with a second call.
- For the open web, the agent picks a general-purpose Actor: RAG Web Browser runs a Google search and reads the top pages, and Website Content Crawler collects a whole site.
Adicionar como fonte preferencialZyla
Zyla’s MCP server pushes the agent toward its catalog before anything else. Its instructions list live data such as flights, weather, stocks and social profiles, and tell the agent not to say it lacks real-time access before trying the catalog.
- The agent describes its need, and Zyla returns ranked APIs. The server tells the agent to take Zyla’s ranking as it is.
- Every API comes from a different provider, so a second source means a new search, new parameters and a new response format. One API writes a month as “Oct/26”, and one Facebook API returns follower counts rounded the way the page shows them, such as “10M”.
Agentic data benchmark methodology
Questions:
We wrote 200 questions, each with one checkable answer:
- 100 on open data sources: 10 each from SEC filings, SEC financial statements, ClinicalTrials.gov, the US App Store, GitHub, NVD, PyPI, PubMed, Crossref and BLS. Records were picked at random, and each answer was computed from the source’s own API.
- 100 on protected domains: 25 each on Facebook, YouTube, Instagram and X. These are the top four domains in Tranco that hold public records and that scraper providers return as structured data. We skipped domains with nothing to ask about, such as fbcdn.net, an image server, and microsoft.com, a corporate site. Every tested catalog lists a source for all four platforms. The questions cover official organization accounts. The answer key comes from the public page, fetched logged out through Zyte, which no tested product uses.
Both sets include one-value, two-value and count-or-compare questions, plus traps about records that do not exist.
We asked every candidate question to the model with no tools and removed those it answered correctly from memory. We also removed questions with missing data or more than one possible answer. Each question names the record and its source, such as ClinicalTrials.gov or the account’s URL, but never a product. A short pilot before each main run checked the wording and the scoring; its results are not reported.
Setup:
- Every product ran on Claude Haiku 5.5 through OpenRouter, at high reasoning effort.
- The agent reached each product through the product’s own MCP server and had no other tool. The server’s own instructions were added to the system prompt, as an MCP client does.
- Firecrawl ran twice, with and without Alexandria. Claude’s web search allowed up to three searches per question and had no page reader.
- Each question was limited to 12 tool calls, 15 minutes and $0,50 in data provider charges. Provider caches were off wherever the product offers a switch.
- Each data layer answered every question 3 times, and Claude’s web search answered once. That makes 1.300 runs per set: 4 data layers × 100 questions × 3, plus 100 Claude runs.
Scoring:
- An answer counted if it matched the answer key captured before or after the run.
- Social media counts could be off by one display step (±0.1M for “1.2M”). Rounded counts such as Facebook’s “10M” for 10.838.386 counted as wrong.
- A run was correct only if every value was right and appeared in the tool output, so a remembered value did not count. Claude’s web search returns no source text, so its answers were checked against the answer key alone.
- Cost is the LLM cost plus the data provider cost, per 100 questions. Time is wall-clock time per question, and tokens are input plus output; both are averaged over correct runs.
- Jobs that failed because a provider’s server was down before the first model call were rerun. Errors during a tool call stayed in the results.
Cite este benchmark
Escolha o formato adequado ao local onde você vai publicar. Colar a versão com link no seu CMS preserva o backlink.
@misc{sipi2026,
author = {Şipi, Nazlı},
title = {{Agentic Data Benchmark: Firecrawl, Apify & Zyla}},
year = {2026},
month = nov,
howpublished = {\url{https://aimultiple.com/agentic-data}},
note = {AIMultiple. Acessado em 13 novembro 2026}
}
Seja o primeiro a comentar
Seu endereço de e-mail não será publicado. Todos os campos são obrigatórios. Os comentários são deixados em seu idioma original.