Large-Scale Web Scraping: 7 Providers Benchmarked
We ran two benchmarks against live websites, from 5 to 5,000 concurrent requests. The first sent 260,000 requests through four web unblockers across the Tranco top 10,000 domains, plus a markdown extraction test on 10,000 URLs. The second fetched 65,000 product and search pages from each of five scraping providers across 100 e-commerce domains.
E-commerce scraper benchmark
Metrics explained
Content-verified success rate: the share of requests that returned the expected content, confirmed against a CSS selector or a structured JSON field carrying the target data. A block page, a captcha or an empty shell scored zero even with an HTTP 200 status.
Median response time (P50): the typical request, in seconds, on the horizontal axis. The tooltip also carries P90, the slowest 10% of requests.
Concurrency: the number of requests in flight at the same time, selectable through the filter buttons. The 5,000 level stresses a provider’s rate limits and account-level ceilings rather than its extraction quality.
Average: throughout this article, the e-commerce average is the mean of the 5- and 100-concurrency tiers. The 5,000 tier is excluded because two of the five providers ran it.
Pricing details and success rates for product and search pages are in the e-commerce scraper benchmark.
Web unblocker benchmark
Metrics explained
Mean completion time: the arithmetic average end-to-end time for successful requests, in seconds, on the horizontal axis. The e-commerce chart plots a median on its horizontal axis, and this one plots a mean, so the two axes carry different statistics.
The markdown extraction results, the pricing table, and the per-provider reviews are in the web unblocker benchmark.
Large-scale scraping benchmark findings
Across 100 e-commerce domains, the highest content-verified success rate was 76.0% and the lowest 59.4%, both averaged over the 5- and 100-concurrency tiers. Across the Tranco top 10,000 domains, the four unblockers ranged from 88% to 94%. These are separate tests against different target sets, different page types and different provider sets, so the two ranges do not combine into one score.
Success peaks at 100 concurrent requests
Every provider in the e-commerce benchmark scored higher at 100 concurrent requests than at 5: Bright Data 73.7% to 78.3%, Zyte 71.1% to 73.0%, Apify 67.7% to 72.5%, Nimble 56.6% to 67.3%, Decodo 55.6% to 63.1%. In the unblocker benchmark, Bright Data peaked at 500 concurrent requests with 95% and Zyte at 500 with 93%.
The decline reaches 5,000, with Bright Data falling to 71.0% and Nimble to 56.0%. So the curve has a middle band rather than a monotonic slope, and a scraper tuned to a low-concurrency setting yields a measurable success rate. The mechanism is not something these benchmarks isolate. Both the 5-request and 5,000-request runs used the same URLs and the same validation, so the difference lies in provider-side behavior that the tests observed rather than in explanations.
Listing pages fail more often than product pages
All five e-commerce providers returned the expected content less often on search and listing pages than on product pages. The gap runs from 4.6 points at Zyte to 14.9 points at Decodo.
Product pages carry a single item with structured fields such as title, price, SKU, and images. Search and listing pages return many items from a query or a category, often paginated and rendered after the initial HTML. For a crawl plan, seeding from a product sitemap or a set of product IDs yields a closer-to-product-page rate than paginating through search results.
The median response time does not predict the tail
Zyte recorded a longer median than Bright Data, 20.9s against 16.2s, and a shorter tail, 52s against 62s. Apify’s tail reached 116s against a 45.7s median, and Decodo’s 23s against 7.0s.
A per-request timeout set from the median therefore cuts a different share of traffic at each provider than one set from P90. A timed-out page also carries a cost, since failed requests count toward the spend but not toward the page total.
Two providers completed a run at 5,000 concurrent requests
In the e-commerce benchmark, Bright Data held 71% and Nimble 56% at 5,000 parallel requests. The other providers were limited by account-level concurrency or credit ceilings at that load rather than by scraping capability, so no result is published for them at that tier. In the unblocker benchmark, Bright Data carries a 5,000-concurrency result at 92% and Nimble at 88%. No 5,000-concurrency figure is published for Zyte or Firecrawl.
Dedicated scrapers cover at most 59 of 100 domains
Providers differ in how they extract data. Some ship a pre-built scraper per marketplace that returns the site’s structured fields, others apply one engine to any site.
At the broadest catalog in the set, 41 of the 100 domains still fall to a universal engine. A fixed target list therefore carries long-tail work at every provider tested, which is an argument for checking a specific domain list against a vendor’s scraper catalog before signing. The metadata-field counts come from two providers and describe those two products, not the two extraction modes in general.
Which layer matches the target list
Three layers sit between a scraper and a target site, and the target list decides which one carries the work.
A scraper API returns parsed fields for a site it already supports. A web unblocker returns the page and leaves parsing to the caller. Residential proxies return a connection and leave both anti-bot handling and parsing to the caller. Scraping browsers sit alongside the unblocker layer, adding programmatic browser control for pages that need interaction.
A known, fixed target list is the case the e-commerce benchmark measures. Such a list holds domains that change rarely, per-site page types worth learning, and a recurring refresh. Dedicated scrapers pay off here to the extent the catalog covers the list, which the coverage table above bounds at 59 of 100 domains. The remainder needs a universal-engine path planned from the start.
An open, long-tail target list is the case the unblocker benchmark measures. Such a list holds domains not known in advance, no per-site parser worth writing, and one generic engine paired with generic extraction. The Tranco top 10,000 is a general-popularity list rather than a list of obscure sites, so it stands in for target-list heterogeneity rather than for target-list obscurity.
Output format cuts across both cases. In the markdown extraction test, providers returned clean text rather than raw HTML.
Markdown extraction test, 10,000 URLs. Every provider present in both tests recorded a lower rate here than on the HTML test, which ran on 260,000 requests over a different URL set. Exa appears in this test alone because it returns no raw HTML documents.
Large language models now handle parsing that once needed hand-written pattern matching, and providers bundle that step into their offering. LLM parsing introduces its own failure mode. Automatically parsed fields need testing against known values, because a hallucinated price reads as valid data. AI web scraping tools covers that category.
Failure modes at large scale
Anti-bot defenses and dynamic pages
Anti-bot systems stack IP reputation, request-header checks, JavaScript challenges and captcha gates, and a page that assembles its content after script execution adds a second surface to clear before any extraction runs.
Clearing a JavaScript challenge means running a browser session rather than a plain HTTP request, and that costs the provider roughly 5 to 10 times more compute. Providers price and pace rendered requests differently as a result.
Account-level concurrency and credit ceilings
At 5,000 parallel requests in the e-commerce benchmark, three of the five providers returned no completed run. The limit reported there was account-level concurrency or credit ceilings at that load rather than scraping capability.
Silent failures and data accuracy
Success in both benchmarks was counted only when the expected content was present in the response, checked against a ground truth defined before the run. An HTTP 200 carrying a block page, a captcha or an empty shell scored zero, so these figures run below what a status-code count produces on the same traffic. A pipeline that checks status codes alone reports success while collecting block pages.
Legal risk
Large-scale collection attracts attention from security teams, and litigation follows when a portion of it touches data behind a login or personal data. Google sued SerpApi over scraping copyrighted content that appeared in its public search results.1 Web scraping legality covers the boundaries in more detail.
Large-scale scraping benchmark methodology
Web unblocker methodology
Four providers, the Tranco top 10,000 domains, 260,000 requests, concurrency 5, 100, 500 and 5,000. Domains were filtered for dead servers, infrastructure-only endpoints and sites with no crawlable content, then screened against public blocklists, a domain-authority threshold and a keyword list. Domains retaining at least three crawlable URLs went forward. Test splits used distinct, non-overlapping URL sets.
Success ran through a staged validation, starting with a bot-page signature pre-filter, then a ground-truth CSS selector match, then text matching on the raw and tag-stripped body, with fallbacks for hashed class names, utility-CSS escaping and character-encoding corruption. A separate markdown extraction test covered 10,000 URLs. One limitation applies. No 5,000-concurrency figure is published for Zyte or Firecrawl.
E-commerce scraper methodology
Five providers, 100 e-commerce domains, 65,000 product and search pages each, concurrency 5, 100 and 5,000. The reported average is the mean of the 5- and 100-concurrency tiers, because Bright Data and Nimble were the two providers that completed a full run at 5,000.
Domains were selected from the top 15 countries by GDP, excluding Russia and China, plus category leaders taken from SimilarWeb and Semrush category lists. Product URLs came from site sitemaps, category crawls and site-restricted search, and search URLs from queries generated out of each domain’s own category names and product slugs, in the site’s own language. Every URL was discovered without fetching the target page through a benchmarked provider. Where a provider offered a dedicated e-commerce scraper for a domain, that scraper ran. Otherwise, the provider’s general unblocker ran. All providers ran from the same server location. Cost was recorded as spend per 1,000 successful product pages, with failed requests counted toward spend.
Add as preferred sourceProviders tested
What is large-scale web scraping?
Large-scale web scraping is the automated collection of web data at a volume where infrastructure choice starts to dictate architecture rather than following from it.
The observable transitions mark the range. At 10,000 requests a month, the unblocker list prices run from $0 at Exa and $7.50 at Bright Data to $99 at Firecrawl. At one million a month the same providers run from $749 to $4,080. At 5,000 concurrent requests, three of the five e-commerce providers produced no completed run. Volume and concurrency, not the scraping technique, are what change at this scale.
Conclusion
The highest content-verified success rate measured across 100 e-commerce domains was 76.0%, averaged over the 5- and 100-concurrency tiers, and the lowest 59.4%. On the Tranco top 10,000, the four unblockers ran from 88% to 94%. Both figures are content-verified counts, which score a block page returned with an HTTP 200 as a failure, so they sit below what a status-code count reports on the same traffic.
For a known, fixed target list, Bright Data recorded 76.0% average success at $1.52 per 1,000 successful product pages and dedicated scrapers for 59 of 100 domains. For response time on the same list, Decodo recorded a 7.0s median and Nimble 8.9s, at 59.4% and 62.0%. For an open, long-tail list, Bright Data recorded 94% at a 5s mean, and Firecrawl 88% at 4s. For sustained load at 5,000 concurrent requests, Bright Data and Nimble carry published results in both benchmarks.
Further reading
- Web Scraping Roadmap: Insights from 30M Requests
- E-Commerce Scraper Benchmark
- Website Unblockers Benchmarked & Compared
- Benchmarked the Best Web Scraping APIs
- Best & Cheapest Residential Proxies
- Best AI Web Scraping Tools
- Most Common Web Scraping Challenges
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{dogan2026,
author = {Dogan, Sedat and Sarı, Ekrem},
title = {{Large-Scale Web Scraping: 7 Providers Benchmarked}},
year = {2026},
month = jul,
howpublished = {\url{https://aimultiple.com/large-scale-web-scraping}},
note = {AIMultiple. Retrieved July 30, 2026}
}- Has 20 years of experience as a white-hat hacker and development guru, with extensive expertise in programming languages and server architectures.
- Is an advisor to C-level executives and board members of corporations with high-traffic and mission-critical technology operations like payment infrastructure.
- Has extensive business acumen alongside his technical expertise.
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.