Web Data Scraping Benchmarks: Providers are now ranked only if they were tested in at least 80% of the benchmarks. Version updated to v1.5.
Web Data Scraping Benchmarks
v1.5400,000+ requests across 10,000+ most popular domains used to measure 5 providers' success rates, response times and metadata coverage. See AIM Web Data Scraping Index Methodology for further details.
Leaderboard
Filter by tool type and must-have requirements. Sort any column.
# | Model | Index | Social media | Search engines | Video & streaming | Reviews | Food delivery | App stores | Real estate | Job postings | Google Maps | Web unblocker |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Bright Data | 91.7 | 86.8 | - | 98.9 | 78.0 | 94.3 | 99.8 | 88.5 | 92.1 | - | 94.8 |
| 2 | Zyte | 88.2 | 99.3 | 72.1 | 99.6 | 65.0 | 82.2 | 99.5 | 65.6 | 64.8 | 99.9 | 95.4 |
| 3 | Nimble | 82.8 | 82.0 | 50.7 | 97.7 | 52.4 | 84.9 | 99.7 | 86.0 | 71.1 | 98.9 | 95.6 |
| 4 | Oxylabs | 81.0 | 91.6 | 97.4 | 99.7 | 56.2 | 90.8 | 99.6 | 71.1 | 64.6 | 94.4 | - |
| 5 | Decodo | 74.1 | 81.9 | 84.7 | 99.5 | 35.9 | 90.0 | 99.7 | 74.8 | 64.5 | - | - |
Cost vs. performance
Web scraper cost across our 100 e-commerce domain benchmark, and web unblocker cost across 10,000 domains. Volume-adjusted per successful request.
Avg is the effective per-1k rate we paid in our benchmark, volume-adjusted with each provider's cheapest available plan. Min is the cheapest tier each provider offers. Max is the most expensive tier.
Web unblocker
Web scraper
Provider profile
Price per successful page, completion time, metadata depth, domains returning structured JSON and AI readiness for each provider.
# | Model | Domains with JSON | Avg metadata fields | $ per 1k | Completion time (s) | AI ready |
|---|---|---|---|---|---|---|
| 1 | Bright Data | 24 | 33 | 1.5 | 22.8 | 79 |
| 2 | Zyte | 0 | 5 | 3.0 | 13.0 | 60 |
| 3 | Nimble | 6 | 5 | 1.0 | 14.6 | 70 |
| 4 | Oxylabs | 12 | 6 | 2.0 | 18.0 | 71 |
| 5 | Decodo | 6 | 9 | 2.4 | 20.1 | 64 |
Behavior under load
Success rate tracked at 1, 100, 500 and 5,000 parallel requests.
Anti-bot landscape
Which vendor guards your targets predicts your success rate.
Web unblocks across the top 10k domains
Scraping APIs across the top 10k domains
Domain coverage by provider
Which of the Tranco top 100 domains each provider can scrape, and for which of them it also returns structured JSON. Scraping support comes from our web unblocker benchmark; JSON support comes from a separate test of parsed output on the same domains. Domains are sorted by coverage, so the ones most providers support appear at the top.
| Domain | |||||||
|---|---|---|---|---|---|---|---|
| google.com | |||||||
| chatgpt.com | |||||||
| bing.com | |||||||
| amazon.com | |||||||
| facebook.com | |||||||
| tiktok.com | |||||||
| youtube.com | |||||||
| baidu.com | |||||||
| googlevideo.com | |||||||
| yandex.ru | |||||||
| apple.com | |||||||
| github.com | |||||||
| instagram.com | |||||||
| microsoft.com | |||||||
| office.com | |||||||
| twitter.com | |||||||
| wikipedia.org | |||||||
| yahoo.com | |||||||
| youtu.be | |||||||
| zoom.us | |||||||
| Domains scraped | 68 | 12 | 6 | 73 | 67 | 10 | 13 |
✓ The provider scrapes the domain. ✅ The provider scrapes the domain and returns structured JSON. ✕ The provider does not scrape the domain. Of the providers in this table, only Bright Data, Nimble and Zyte ran in the web unblocker benchmark; for the other providers the table shows JSON support only.
Web data providers markdown output performance
Markdown extraction success rate and completion time from our web unblocker benchmark, and the output formats each provider returns.
| Provider | Markdown | HTML |
|---|---|---|
This table shows which output formats each provider supports. It is not a test result: the markdown test above covered Bright Data, Exa, Firecrawl and Nimble.
Success rate by page type and concurrency
Product pages and search pages behave differently on the same domain.
Frequently asked questions
AIM Web Data Scraping Index Methodology
The index re-uses the raw results of every web data benchmark we published. A benchmark counts if it tests web data providers and records the outcome of each request. A provider's index score is the average of its success rates across the benchmarks it ran in. A provider is ranked only if it was tested in at least 80% of the benchmarks, the same rule the SDG Index uses. Providers below that still appear in the charts, without a rank. Every benchmark carries equal weight, so a 1,400-request study counts as much as a 260,000-request one. Success means the response returned the target page with the content we asked for. We check that against bot pages and empty shells, so a 200 response with a challenge page counts as a failure. If a provider was never tested in a benchmark, we show a dash instead of a zero.
Targets come from the Tranco list, which ranks domains by averaging several traffic rankings. We remove dead, adult, gambling and malicious hosts before testing. Domains are grouped into categories: e-commerce, social media, search engines, video and streaming, travel, reviews, food delivery, app stores, real estate, job postings, news and publishing, developer and SaaS, finance, and government and education. Domain coverage and structured JSON support are checked on the top 100. Cost is per 1,000 successful pages, not per 1,000 requests sent. We take the cheapest plan a provider offers at that volume and divide by the success rate we measured. Completion time is how long one request takes from start to finish. Metadata fields are the parsed fields a provider returns when it answers with JSON. AI ready is the share of responses that came back as clean markdown. The anti-bot results come from two of our runs, the unblocker benchmark across 10,000 domains and the e-commerce benchmark across the top 100 domains, with each domain labelled by the vendor guarding it. Whiskers are 95% confidence intervals.
Raw results of 330.8 thousand data points from the AIMultiple benchmarks this index is built from. Download them as a ZIP file containing 12 CSV files and a README.
Want the granular data behind it? Join Premium
Changelog
16 updates- v1.5
Web Data Scraping Benchmarks: Providers are now ranked only if they were tested in at least 80% of the benchmarks. Version updated to v1.5.
- v1.4
Web Data Scraping Benchmarks: Average metadata fields now count only responses that returned parsed JSON; raw HTML answers no longer count as zero fields. Version updated to v1.4.
- v1.3
Web Data Scraping Benchmarks: The index is now built from the raw results of every published web data benchmark, each weighted equally. Version updated to v1.3.
- v1.2
Web Data Scraping Benchmarks: Each benchmark is now scored on its own, and providers with too few samples are left out. Version updated to v1.2.
- v1.1
5 providers entered the index: Bright Data, Decodo, Nimble, Oxylabs, Zyte.
- v1.1
Web Data Scraping Benchmarks: Providers are now scored per benchmark category and per tool type (scraping API or web unblocker). Version updated to v1.1.
- v1.0
Ethical web data benchmark: Added a Methodology limitations section at the end of the benchmark.
Explore Web Data Scraping Benchmarks
Best Apify Alternatives: Pricing, Features Compared
Scraping ToolsBenchmarkNov 19Apify runs web scrapers called Actors and offers a marketplace with thousands of prebuilt Actors for sites like Amazon, Google Maps, and Instagram. We tested its alternatives in our scraping benchmarks and compared their features and pricing. We calculated each scraper’s effective cost, the price per successful request, from the success rates in our e-commerce…
- Web Data ScrapingBenchmarkNov 19
Tavily Alternatives: Tested & Compared
Tavily is a web search API for AI agents that returns search results, extracted page content and an optional generated answer in one call. We tested its alternatives and compared their features and pricing. We benchmarked 24 configurations of Tavily, Brave, Exa, Firecrawl, Parallel and Perplexity on 120 questions: 50 retrieval queries and 70 answer…
- Web Data ScrapingOpen World EvaluationNov 19
Best Exa Alternatives: Features and Pricing Compared
Exa is a search API that returns web results and page contents in a format LLMs can use directly. We tested its alternatives and compared their features and pricing. Fast and deep search modes: the search API lets you trade speed for depth or relevance. Cited report: searches many sources and returns a written report…
- E-commerce ScrapingInsightNov 19
Top 6 Food Delivery Scrapers: Benchmark & Use Cases
We benchmarked 6 web scraping providers to see how they handle food delivery data scraping, sending 12,000 requests in total across the top 4 food delivery platforms, and measured success rate, completion time, and metadata coverage. See the benchmark methodology section for more details on the testing process. Different platforms expose different layers of data,…
- Scraping ToolsBenchmarkNov 18
Best Scraping Browsers
Scraping browsers are remote browsers that get past CAPTCHAs and bot checks, and AI agents can control them through MCP servers or CLIs. We measured the success rate and speed of Bright Data’s and ZenRows’ scraping browsers on 85 agent tasks across 28 websites. For more details on how we tested, see our benchmark methodology.…
- Social Media ScrapingBenchmarkNov 16
Top 7 Video Scrapers: Tested & Ranked
We tested the top 7 video scraping providers to see how they handle video metadata on the top video platform, totaling 6,000 requests, and measured their success rate, response time, and metadata fields. To see how we calculated these metrics, read video scraping benchmark methodology. Different providers return different amounts of metadata for the same…
- Social Media ScrapingInsightNov 16
Best Facebook Scrapers: Apify, Bright Data & Decodo
We benchmarked six web scraping APIs with 7,500 Facebook requests on 1,500 URLs: 500 profiles, 500 posts and 500 reels. For every request we measured success rate, completion time, and the number of parsed metadata fields returned. Bright Data returned JSON for every Facebook page type. Apify returned JSON for profiles and posts, SerpApi and…
- Social Media ScrapingBenchmarkNov 16
Best Twitter (X) Scrapers: Benchmarked
Four web scraping APIs each received the same 1,000 X URLs, 500 profiles and 500 posts, for 4,000 requests in total. We recorded the success rate, completion time and number of parsed metadata fields of every response. See our benchmark methodology for more details. structured JSON, HTML only, not supported. When scraping X, you get…