Web Crawler Benchmark to Feed Websites to AI

We benchmarked four crawl APIs across three domains of varying difficulty at three max depth levels (5, 10, 20) with a 1,000-page limit, measuring crawl coverage, execution time, link discovery, markdown link quality, and title extraction accuracy.

If you aim to:

• Turn web pages into structured data, see our guide on web scraping.
• Crawl entire websites, read on.

Web crawlers benchmark

You can read our benchmark methodology.

Average crawled pages vs cost per 1,000 pages

Crawled pages across domains by max depth

Firecrawl consistently crawled around 100 pages on theregister.com regardless of max depth, approximately 90 pages on entrepreneur.com across all depth levels, and only around 30 pages on amazon.com, likely due to Amazon’s aggressive bot protection. Notably, increasing max depth had virtually no impact on the number of pages Firecrawl was able to crawl across any domain.

Apify demonstrated the most consistent performance, reaching the maximum crawl limit of 1,000 pages on every domain at every depth level without any apparent difficulty, even on heavily protected sites like Amazon.

Cloudflare showed inconsistent behavior across tests:

• On theregister.com at max depth 5, it crawled only 100 pages, but at max depth 20 it reached nearly 1,000 pages.
• As we observed in earlier tests, Cloudflare occasionally crawls just 1 page and then terminates the job entirely. We confirmed this is not a caching issue (cache was disabled) and tested with wait times between runs of up to 1 minute, but the behavior persisted. At max depth 10 on theregister.com, this exact issue occurred, Cloudflare crawled only 1 page before stopping.
• On entrepreneur.com, Cloudflare crawled 780 pages at depth 5, increased to 885 at depth 10, but then dropped sharply to just 172 pages at depth 20. This drop may be related to Cloudflare’s crawl scheduler deprioritizing or timing out deeper link chains, or it could reflect an internal concurrency limit that causes the job to terminate prematurely when the crawl frontier grows too large at higher depths.
• On amazon.com, Cloudflare crawled 905 pages at depth 5, but the number steadily declined as max depth increased, dropping to 809 at depth 10 and 795 at depth 20, suggesting that deeper crawl configurations may cause Cloudflare to spend more time on link discovery overhead rather than actual page retrieval.

Nimble reached or approached the 1,000-page limit on theregister.com across all depth levels (1,000 / 1,000 / 999). On entrepreneur.com, it crawled 1,000 pages at depth 5 but showed slight drops at higher depths (896 at depth 10, 983 at depth 20), possibly due to its 7-hour timeout being reached before completing the full crawl at deeper levels, all Nimble runs ended with a timeout status. Amazon proved more challenging:

• At depth 5 it managed only 319 pages, but at depth 10 it jumped to 988 pages, then dropped to 906 at depth 20
• This inconsistency likely reflects the combination of Amazon’s bot protection mechanisms and Nimble’s timeout constraints, where deeper crawls take longer to process each page and may encounter more anti-bot challenges along the way

Execution time across domains by max depth

Firecrawl was the fastest provider across all domains, completing crawls in under 5 minutes, typically between 75-265 seconds. This speed comes at the cost of coverage, as Firecrawl also crawled the fewest pages. Essentially, it finishes quickly because it stops early.

Apify took around 2,200-2,400 seconds (~40 minutes) on theregister.com regardless of depth. On entrepreneur.com and amazon.com, execution times were significantly longer at 8,300-15,900 seconds (2-4 hours), reflecting the larger and more complex site structures. Despite the longer times, Apify consistently reached the 1,000-page limit, making it the most reliable in terms of coverage-to-time ratio.

Cloudflare showed timing that mirrors its inconsistent crawl counts:

• On theregister.com at depth 10, it completed in just 1 second, because it only crawled 1 page before stopping.
• On entrepreneur.com at depth 20, it finished in 10 seconds after crawling only 172 pages.
• When Cloudflare does complete a full crawl, times range from 3,500 to 25,200 seconds.
• As max depth increases, Cloudflare appears to prioritize reaching deeper pages over breadth, crawling fewer pages but completing faster. On amazon.com, execution time dropped from 25,200 seconds (timeout) at depth 5 to just 5,660 seconds at depth 20, while crawled pages also decreased from 905 to 795. This suggests Cloudflare’s crawler shifts its strategy at higher depths, spending less time on broad discovery and more on deep traversal.

Nimble hit the 7-hour timeout (25,200 seconds) on every single run across all domains and depth levels. This is notable because in our earlier quick tests with max depth 1, Nimble completed without timing out. In the full benchmark with depths of 5-20 and a 1,000-page limit, it consistently ran until the timeout was reached. Despite this, Nimble still managed to crawl a high number of pages in most cases (~900-1,000 on theregister.com and entrepreneur.com), meaning it is actively crawling throughout the 7 hours but simply never signals completion.

Link text fill rate across providers by max depth

To assess markdown output quality, we measured what percentage of links in each provider’s markdown contain anchor text, the clickable text portion of a link. A missing anchor text (e.g., [](/about) instead of [About Us](/about)) means the crawler failed to extract the link’s label.

• Nimble: 100% across all depths
• Cloudflare: 91-94%
• Firecrawl: 90%
• Apify: 77-78% ,  roughly 1 in 5 links missing anchor text

Crawl depth had minimal impact on fill rates for any provider, suggesting this is a characteristic of each provider’s parsing engine rather than a crawl setting.

Link text fill rate across providers by domain

Looking at fill rates across different domains reveals how site complexity affects each provider’s link extraction quality.

• Nimble maintained 100% across all domains.
• Apify showed the most variation, 89% on amazon.com but dropping to 66% on entrepreneur.com, meaning one-third of its links on that site were missing anchor text. This suggests Apify struggles more with content-heavy sites that have complex navigation structures.
• Firecrawl performed best on theregister.com (98%) but dropped to 81% on entrepreneur.com, following a similar pattern to Apify.
• Cloudflare was the most consistent after Nimble, staying between 89-94% regardless of domain.

Entrepreneur.com proved the most challenging domain for link text extraction, both Apify (66%) and Firecrawl (81%) had their lowest scores there, likely due to the site’s heavy use of nested navigation menus and dynamic content elements that are harder to convert cleanly into markdown.

Total links in markdown output across domains by max depth

Link count variance across providers was consistently high (74-97%), indicating that providers extract very different numbers of links from the same pages. To get a more detailed view of this disparity, we measured the total markdown link count per provider.

• Apify returned the most links overall, particularly on amazon.com with over 420K links at depth 5 (~423 per page). On entrepreneur.com it stabilized around 63K regardless of depth. Its output includes ad trackers and tracking pixels alongside page content links.
• Cloudflare peaked at 303K on entrepreneur.com at depth 10 but dropped to 53K at depth 20. On the same entrepreneur.com homepage, Cloudflare extracted 434 links compared to Apify’s 143, capturing full navigation menus and submenus.
• Firecrawl consistently returned 5-9K links across all configurations, limited by its low page count.
• Nimble returned 3-40K links total, averaging 5-28 links per page compared to 60-420 for other providers. On entrepreneur.com’s homepage, Nimble returned 13 links versus Cloudflare’s 434, limited to main article headlines. Its 100% fill rate reflects that the links it did include all had anchor text, rather than indicating comprehensive link coverage. Nimble does not output standard markdown links. Its count includes escaped HTML links found within the markdown output.

Title present rate across providers

Title similarity across providers showed less than 1% deviation across all tests and domains, confirming that when providers do extract a title, they consistently return the same result. Title present rate also remained between 98-100% across all max depth levels, showing that crawl depth has no meaningful impact on title extraction.

When broken down by domain, some differences emerged: 

On entrepreneur.com and theregister.com, most providers achieved 99-100% title present rates. Amazon.com was the only domain where meaningful differences appeared,  Firecrawl dropped to 93% and Nimble to 95.9%, while Apify maintained 99.6%. This aligns with Amazon’s heavier bot protection, which can block or distort page responses, causing some providers to return pages without extractable titles.

Features of the best web crawler APIs

Decodo provides crawling through /crawl, an endpoint of its Web Data API. All Web Data API endpoints run under one API and one API key. A /crawl job collects up to 10,000 pages and returns each page as Markdown or HTML.

• Users set the maximum depth, filter paths with regular expressions and keep the crawl on one host, its subdomains or all domains.
• The crawler uses the sitemap, page links or both, and requests pages from a chosen country and locale.
• A maximum duration stops the job and keeps the pages collected so far.
• A page that fails without JavaScript gets one more try with JavaScript rendering.

The /map endpoint lists up to 147,000 URLs from a page and the site’s top-level sitemap without fetching content, so a crawl can target selected paths only. The free plan includes 10,000 requests per month without a credit card.

Apify crawls websites through Website Content Crawler, an Actor on the Apify platform. By default it fetches static pages over plain HTTP and switches to a headless browser for pages that need JavaScript. Users can also run it in browser-only or HTTP-only mode.

• Users set start URLs, maximum depth, maximum pages and URL patterns to include or exclude, and can add sitemaps for discovery.
• A “modified since” date limits recrawls to changed pages.
• A CSS selector removes navigation, headers and footers before conversion.
• The crawler returns text, Markdown, HTML and screenshots, and it can download linked PDF, Word and Excel files.

Apify connects the output to LangChain, LlamaIndex, Pinecone and Qdrant, and offers the crawler as an MCP tool.

Cloudflare offers a /crawl endpoint in its Browser Rendering service. One job covers up to 100,000 pages and runs for up to seven days, and results stay available for 14 days.

• The crawler finds pages through sitemaps, page links or both, and filters URLs with wildcard patterns.
• It renders pages in a headless browser by default and can fetch plain HTML without JavaScript instead.
• Output is HTML, Markdown or JSON.

The crawler obeys robots.txt, including crawl-delay, and identifies itself with a fixed user agent. Cloudflare states that it does not bypass CAPTCHAs or bot protection.

Firecrawl’s /crawl endpoint discovers and scrapes the subpages of a starting URL and returns Markdown, HTML or JSON for each page.

• Users set a page limit, a discovery depth and regex path filters.
• The crawl can extend to parent paths, subdomains or one hop of external links.
• The crawler combines the sitemap with page links, or uses only one of the two.
• A prompt parameter turns a plain-language instruction into crawl settings.

Results arrive by polling, per-page webhook events or an SDK watcher, and they stay available for 24 hours.

Nimble Crawl runs each crawl as a background job of up to 10,000 pages and 20 link hops, and returns HTML or Markdown for each page.

• Users filter paths with regular expressions and choose whether to follow subdomains.
• The crawler uses the sitemap, page links or both as the source.
• URLs that differ only in query parameters can count as one page.
• Each crawl can run from a chosen country and locale.

A status call reports how many pages are pending, completed or failed, and each page’s content comes from its own result call. Callbacks fire when the crawl starts, for each page and at the end.

Web crawlers benchmark methodology

We tested four crawl APIs (Apify, Nimble, Cloudflare, Firecrawl) on three domains of varying difficulty: amazon.com (heavy bot protection), entrepreneur.com (complex content site), and theregister.com (news site).

Shared configuration

All providers received identical core settings to ensure a fair comparison:

• Sitemap: Disabled,  providers must discover pages through HTML links only
• External links: Disabled,  crawlers stay within the target domain
• Subdomains: Enabled, subdomain pages are followed (e.g., india.entrepreneur.com)
• JavaScript rendering: Enabled, all providers use a headless browser
• Cache: Disabled 
• Page limit: 1,000 pages per run
• Timeout: 7 hours (25,200 seconds)
• Rate limit handling: 20-second wait with up to 3 retries on HTTP 429

Each provider was tested at three max depth levels (5, 10, 20) across all three domains, totaling 36 crawl runs. Providers were tested sequentially (not in parallel), each combination was run once, and crawl status was polled every 1 second.

Apify was configured with the website-content-crawler actor using Playwright/Firefox as its headless browser. Subdomain access was controlled via glob patterns, and Apify’s built-in proxy was used for all requests.

Nimble, Cloudflare, and Firecrawl were configured using their respective REST APIs with the shared settings described above. No additional provider-specific configurations were applied beyond the standardized parameters.

For Cloudflare, we used the Workers Paid plan. The reported cost reflects what we spent to crawl 1,000 pages under this plan. Cloudflare charges based on browser rendering time rather than page count.

For Firecrawl, we used the Hobby plan. The reported cost is the prorated amount for 1,000 credits out of the credits provided in this plan. Effective per-page cost varies depending on the plan tier and whether extra credit packs are purchased.

Cite this benchmark

Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.

@misc{dilmegani2026,
  author = {Dilmegani, Cem and Şipi, Nazlı},
  title  = {{Web Crawler Benchmark to Feed Websites to AI}},
  year   = {2026},
  month  = oct,
  howpublished    = {\url{https://aimultiple.com/web-crawler}},
  note   = {AIMultiple. Retrieved October 22, 2026}
}

Changelog

9 updates

All9General8Methodology1

202620242023

1. Apr 7

   Removed the industry trends, crawler types, and web crawling examples sections.

2. Apr 6

   Added a benchmark of four crawl APIs across three domains and three depth levels, with charts on coverage, speed, links, and titles.

3. Apr 6

   Added a methodology section detailing the crawl setup across four providers, three domains, and three depth levels.

4. Feb 25

   Added a section, Are web crawlers legal?, to the article.

5. Feb 11

   Updated the "What is a web crawler?" section with new trends and modes of operation.

Cem Dilmegani

Principal Analyst

Follow On

Cem has been the principal analyst at AIMultiple since 2017.

Cem's work at AIMultiple has been cited by leading global publications including Business Insider, Forbes, Morning Brew, and Washington Post, global firms like Deloitte and HPE, NGOs like World Economic Forum, and supranational organizations like European Commission. [1], [2], [3], [4], [5]

Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.

He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.

Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.

View Full Profile

Technically reviewed by

Nazlı Şipi

AI Researcher

Follow On

Nazlı is a data analyst at AIMultiple. She has prior experience in data analysis across various industries, where she worked on transforming complex datasets into actionable insights.

View Full Profile

Comments 1

Share Your Thoughts

Your email address will not be published. All fields are required. Comments are left in their original language.

Name

Email Address

Comment

0/450

Post Comment

Aggeliki

Jan 12, 2022 at 16:15

Hi Cem, I think there is a misunderstanding regarding the robots.txt role in the crawling context. The web bots can crawl any website when indexing is allowed without having the robots.txt somewhere on their top domain, subdomains and ports and so on. The role of a robots.txt is to keep control of the traffic from web bots so the website is not overloaded by requests.

Reply