Web Data Scraping Benchmarks: Providers are now ranked only if they were tested in at least 80% of the benchmarks. Version updated to v1.5.
Web Data Scraping Benchmarks
v1.5400,000+ requests across 10,000+ most popular domains used to measure 5 providers' success rates, response times and metadata coverage. See AIM Web Data Scraping Index Methodology for further details.
Leaderboard
Filter by tool type and must-have requirements. Sort any column.
# | Model | Index | Social media | Search engines | Video & streaming | Reviews | Food delivery | App stores | Real estate | Job postings | Google Maps | Web unblocker |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Bright Data | 91.7 | 86.8 | - | 98.9 | 78.0 | 94.3 | 99.8 | 88.5 | 92.1 | - | 94.8 |
| 2 | Zyte | 88.2 | 99.3 | 72.1 | 99.6 | 65.0 | 82.2 | 99.5 | 65.6 | 64.8 | 99.9 | 95.4 |
| 3 | Nimble | 82.8 | 82.0 | 50.7 | 97.7 | 52.4 | 84.9 | 99.7 | 86.0 | 71.1 | 98.9 | 95.6 |
| 4 | Oxylabs | 81.0 | 91.6 | 97.4 | 99.7 | 56.2 | 90.8 | 99.6 | 71.1 | 64.6 | 94.4 | - |
| 5 | Decodo | 74.1 | 81.9 | 84.7 | 99.5 | 35.9 | 90.0 | 99.7 | 74.8 | 64.5 | - | - |
Cost vs. performance
Web scraper cost across our 100 e-commerce domain benchmark, and web unblocker cost across 10,000 domains. Volume-adjusted per successful request.
Avg is the effective per-1k rate we paid in our benchmark, volume-adjusted with each provider's cheapest available plan. Min is the cheapest tier each provider offers. Max is the most expensive tier.
Web unblocker
Web scraper
Provider profile
Price per successful page, completion time, metadata depth, domains returning structured JSON and AI readiness for each provider.
# | Model | Domains with JSON | Avg metadata fields | $ per 1k | Completion time (s) | AI ready |
|---|---|---|---|---|---|---|
| 1 | Bright Data | 24 | 33 | 1.5 | 22.8 | 79 |
| 2 | Zyte | 0 | 5 | 3.0 | 13.0 | 60 |
| 3 | Nimble | 6 | 5 | 1.0 | 14.6 | 70 |
| 4 | Oxylabs | 12 | 6 | 2.0 | 18.0 | 71 |
| 5 | Decodo | 6 | 9 | 2.4 | 20.1 | 64 |
Behavior under load
Success rate tracked at 1, 100, 500 and 5,000 parallel requests.
Anti-bot landscape
Which vendor guards your targets predicts your success rate.
Web unblocks across the top 10k domains
Scraping APIs across the top 10k domains
Domain coverage by provider
Which of the Tranco top 100 domains each provider can scrape, and for which of them it also returns structured JSON. Scraping support comes from our web unblocker benchmark; JSON support comes from a separate test of parsed output on the same domains. Domains are sorted by coverage, so the ones most providers support appear at the top.
| Domain | |||||||
|---|---|---|---|---|---|---|---|
| google.com | |||||||
| chatgpt.com | |||||||
| bing.com | |||||||
| amazon.com | |||||||
| facebook.com | |||||||
| tiktok.com | |||||||
| youtube.com | |||||||
| baidu.com | |||||||
| googlevideo.com | |||||||
| yandex.ru | |||||||
| apple.com | |||||||
| github.com | |||||||
| instagram.com | |||||||
| microsoft.com | |||||||
| office.com | |||||||
| twitter.com | |||||||
| wikipedia.org | |||||||
| yahoo.com | |||||||
| youtu.be | |||||||
| zoom.us | |||||||
| Domains scraped | 68 | 12 | 6 | 73 | 67 | 10 | 13 |
✓ The provider scrapes the domain. ✅ The provider scrapes the domain and returns structured JSON. ✕ The provider does not scrape the domain. Of the providers in this table, only Bright Data, Nimble and Zyte ran in the web unblocker benchmark; for the other providers the table shows JSON support only.
Web data providers markdown output performance
Markdown extraction success rate and completion time from our web unblocker benchmark, and the output formats each provider returns.
| Provider | Markdown | HTML |
|---|---|---|
This table shows which output formats each provider supports. It is not a test result: the markdown test above covered Bright Data, Exa, Firecrawl and Nimble.
Success rate by page type and concurrency
Product pages and search pages behave differently on the same domain.
Frequently asked questions
AIM Web Data Scraping Index Methodology
The index re-uses the raw results of every web data benchmark we published. A benchmark counts if it tests web data providers and records the outcome of each request. A provider's index score is the average of its success rates across the benchmarks it ran in. A provider is ranked only if it was tested in at least 80% of the benchmarks, the same rule the SDG Index uses. Providers below that still appear in the charts, without a rank. Every benchmark carries equal weight, so a 1,400-request study counts as much as a 260,000-request one. Success means the response returned the target page with the content we asked for. We check that against bot pages and empty shells, so a 200 response with a challenge page counts as a failure. If a provider was never tested in a benchmark, we show a dash instead of a zero.
Targets come from the Tranco list, which ranks domains by averaging several traffic rankings. We remove dead, adult, gambling and malicious hosts before testing. Domains are grouped into categories: e-commerce, social media, search engines, video and streaming, travel, reviews, food delivery, app stores, real estate, job postings, news and publishing, developer and SaaS, finance, and government and education. Domain coverage and structured JSON support are checked on the top 100. Cost is per 1,000 successful pages, not per 1,000 requests sent. We take the cheapest plan a provider offers at that volume and divide by the success rate we measured. Completion time is how long one request takes from start to finish. Metadata fields are the parsed fields a provider returns when it answers with JSON. AI ready is the share of responses that came back as clean markdown. The anti-bot results come from two of our runs, the unblocker benchmark across 10,000 domains and the e-commerce benchmark across the top 100 domains, with each domain labelled by the vendor guarding it. Whiskers are 95% confidence intervals.
Raw results of 330.8 thousand data points from the AIMultiple benchmarks this index is built from. Download them as a ZIP file containing 12 CSV files and a README.
Want the granular data behind it? Join Premium
Changelog
16 updates- v1.5
Web Data Scraping Benchmarks: Providers are now ranked only if they were tested in at least 80% of the benchmarks. Version updated to v1.5.
- v1.4
Web Data Scraping Benchmarks: Average metadata fields now count only responses that returned parsed JSON; raw HTML answers no longer count as zero fields. Version updated to v1.4.
- v1.3
Web Data Scraping Benchmarks: The index is now built from the raw results of every published web data benchmark, each weighted equally. Version updated to v1.3.
- v1.2
Web Data Scraping Benchmarks: Each benchmark is now scored on its own, and providers with too few samples are left out. Version updated to v1.2.
- v1.1
5 providers entered the index: Bright Data, Decodo, Nimble, Oxylabs, Zyte.
- v1.1
Web Data Scraping Benchmarks: Providers are now scored per benchmark category and per tool type (scraping API or web unblocker). Version updated to v1.1.
- v1.0
Ethical web data benchmark: Added a Methodology limitations section at the end of the benchmark.
Explore Web Data Scraping Benchmarks
- Scraping ToolsBenchmarkNov 6
Top 6 LLM Scrapers: ChatGPT, Perplexity & Gemini
We benchmarked how the top LLM scraper providers, including Bright Data, Oxylabs, and Apify, perform at extracting outputs from LLM platforms such as ChatGPT, Gemini, Perplexity, and Google AI Mode. To ensure reliable results, we ran 1,000 tests per provider, repeating each prompt 10 times for consistency. The top-performing provider is detailed below. Providers missing…
- Web Data ScrapingInsightNov 4
Best AI Web Scraping Tools Benchmarked
We benchmarked top AI web scraping tools with 25,000 requests, 5,000 per tool, across 500 URLs on each of 10 of the most visited domains to see their performance, and also compared how each one leverages AI. See the benchmark methodology section for more details on the testing process. For each domain we asked for…
- E-commerce ScrapingBenchmarkNov 2
eBay Scraping: Top 7 Providers Compared
We benchmarked eBay across 5 web scraping providers, totaling 1,750 requests over search and product pages of 7 eBay country sites, measuring both success rate and end-to-end completion time. You can read more about our benchmark methodology. Bright Data, Apify and SerpApi were the three providers that returned parsed JSON. Field counts came to 59…
- Review ScrapingBenchmarkNov 2
Top 6 Amazon Review Scrapers Compared
To compare how web data scraping providers handle Amazon review extraction, we tested 5 web scraping providers on the same set of Amazon product review URLs, totaling 2,500 requests across all providers. Read our benchmark methodology for more detail on our testing process. Amazon was the most accessible platform in our reviews scraping benchmark. The…
- Scraping ToolsOpen World EvaluationNov 2
Top 11 Open Source Web Crawlers for LLM & AI
Recent advances in generative AI have reshaped what developers need from web crawlers. Agentic crawlers now use natural-language prompts to select links rather than fixed rules, and produce token-efficient markdown natively. At the same time, the classic frameworks for large-scale batch crawling remain irreplaceable for enterprise and research use. Language: Python | License: Apache 2.0…
- Scraping ToolsBenchmarkNov 2
5 Best Google Maps Scraper APIs: Tested & Ranked
To find the best Google Maps scraper, we benchmarked five web scraping providers, Apify, SerpApi, Oxylabs, Zyte and Nimble, by requesting the same 1,000 businesses from each. The businesses span 10 categories across 10 US cities, for 5,000 requests in total. Since all providers exceeded 90% success, we compared completion time: Only SerpApi and Apify…
- Scraping ToolsBenchmarkNov 2
We Tested the Best SERP Scraper APIs
We benchmarked 5 SERP providers using 7,000 live requests across Google, Bing, and Yandex. See which providers came out ahead on success rate and speed: Read the full methodology. We also run live requests every 15 min, with caching disabled, a 60s timeout, and 250+ queries across Google, Bing, and Yandex in the United States.…
- Anti-BlockingBenchmarkOct 27
Top 5 Website Unblockers Benchmarked & Compared
We benchmarked 4 leading web data providers across the top 10,000 domains, running a total of 260,000 requests. Each provider was tested at multiple concurrency levels to measure how they behave under increasing load. We also ran a separate markdown extraction test on 10,000 URLs, with Exa added, to measure how often each provider returns…