Web Data Scraping Benchmarks: Providers are now ranked only if they were tested in at least 80% of the benchmarks. Version updated to v1.5.
Web Data Scraping Benchmarks
v1.5400,000+ requests across 10,000+ most popular domains used to measure 5 providers' success rates, response times and metadata coverage. See AIM Web Data Scraping Index Methodology for further details.
Leaderboard
Filter by tool type and must-have requirements. Sort any column.
# | Model | Index | Social media | Search engines | Video & streaming | Reviews | Food delivery | App stores | Real estate | Job postings | Google Maps | Web unblocker |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Bright Data | 91.7 | 86.8 | - | 98.9 | 78.0 | 94.3 | 99.8 | 88.5 | 92.1 | - | 94.8 |
| 2 | Zyte | 88.2 | 99.3 | 72.1 | 99.6 | 65.0 | 82.2 | 99.5 | 65.6 | 64.8 | 99.9 | 95.4 |
| 3 | Nimble | 82.8 | 82.0 | 50.7 | 97.7 | 52.4 | 84.9 | 99.7 | 86.0 | 71.1 | 98.9 | 95.6 |
| 4 | Oxylabs | 81.0 | 91.6 | 97.4 | 99.7 | 56.2 | 90.8 | 99.6 | 71.1 | 64.6 | 94.4 | - |
| 5 | Decodo | 74.1 | 81.9 | 84.7 | 99.5 | 35.9 | 90.0 | 99.7 | 74.8 | 64.5 | - | - |
Cost vs. performance
Web scraper cost across our 100 e-commerce domain benchmark, and web unblocker cost across 10,000 domains. Volume-adjusted per successful request.
Avg is the effective per-1k rate we paid in our benchmark, volume-adjusted with each provider's cheapest available plan. Min is the cheapest tier each provider offers. Max is the most expensive tier.
Web unblocker
Web scraper
Provider profile
Price per successful page, completion time, metadata depth, domains returning structured JSON and AI readiness for each provider.
# | Model | Domains with JSON | Avg metadata fields | $ per 1k | Completion time (s) | AI ready |
|---|---|---|---|---|---|---|
| 1 | Bright Data | 24 | 33 | 1.5 | 22.8 | 79 |
| 2 | Zyte | 0 | 5 | 3.0 | 13.0 | 60 |
| 3 | Nimble | 6 | 5 | 1.0 | 14.6 | 70 |
| 4 | Oxylabs | 12 | 6 | 2.0 | 18.0 | 71 |
| 5 | Decodo | 6 | 9 | 2.4 | 20.1 | 64 |
Behavior under load
Success rate tracked at 1, 100, 500 and 5,000 parallel requests.
Anti-bot landscape
Which vendor guards your targets predicts your success rate.
Web unblocks across the top 10k domains
Scraping APIs across the top 10k domains
Domain coverage by provider
Which of the Tranco top 100 domains each provider can scrape, and for which of them it also returns structured JSON. Scraping support comes from our web unblocker benchmark; JSON support comes from a separate test of parsed output on the same domains. Domains are sorted by coverage, so the ones most providers support appear at the top.
| Domain | |||||||
|---|---|---|---|---|---|---|---|
| google.com | |||||||
| chatgpt.com | |||||||
| bing.com | |||||||
| amazon.com | |||||||
| facebook.com | |||||||
| tiktok.com | |||||||
| youtube.com | |||||||
| baidu.com | |||||||
| googlevideo.com | |||||||
| yandex.ru | |||||||
| apple.com | |||||||
| github.com | |||||||
| instagram.com | |||||||
| microsoft.com | |||||||
| office.com | |||||||
| twitter.com | |||||||
| wikipedia.org | |||||||
| yahoo.com | |||||||
| youtu.be | |||||||
| zoom.us | |||||||
| Domains scraped | 68 | 12 | 6 | 73 | 67 | 10 | 13 |
✓ The provider scrapes the domain. ✅ The provider scrapes the domain and returns structured JSON. ✕ The provider does not scrape the domain. Of the providers in this table, only Bright Data, Nimble and Zyte ran in the web unblocker benchmark; for the other providers the table shows JSON support only.
Web data providers markdown output performance
Markdown extraction success rate and completion time from our web unblocker benchmark, and the output formats each provider returns.
| Provider | Markdown | HTML |
|---|---|---|
This table shows which output formats each provider supports. It is not a test result: the markdown test above covered Bright Data, Exa, Firecrawl and Nimble.
Success rate by page type and concurrency
Product pages and search pages behave differently on the same domain.
Frequently asked questions
AIM Web Data Scraping Index Methodology
The index re-uses the raw results of every web data benchmark we published. A benchmark counts if it tests web data providers and records the outcome of each request. A provider's index score is the average of its success rates across the benchmarks it ran in. A provider is ranked only if it was tested in at least 80% of the benchmarks, the same rule the SDG Index uses. Providers below that still appear in the charts, without a rank. Every benchmark carries equal weight, so a 1,400-request study counts as much as a 260,000-request one. Success means the response returned the target page with the content we asked for. We check that against bot pages and empty shells, so a 200 response with a challenge page counts as a failure. If a provider was never tested in a benchmark, we show a dash instead of a zero.
Targets come from the Tranco list, which ranks domains by averaging several traffic rankings. We remove dead, adult, gambling and malicious hosts before testing. Domains are grouped into categories: e-commerce, social media, search engines, video and streaming, travel, reviews, food delivery, app stores, real estate, job postings, news and publishing, developer and SaaS, finance, and government and education. Domain coverage and structured JSON support are checked on the top 100. Cost is per 1,000 successful pages, not per 1,000 requests sent. We take the cheapest plan a provider offers at that volume and divide by the success rate we measured. Completion time is how long one request takes from start to finish. Metadata fields are the parsed fields a provider returns when it answers with JSON. AI ready is the share of responses that came back as clean markdown. The anti-bot results come from two of our runs, the unblocker benchmark across 10,000 domains and the e-commerce benchmark across the top 100 domains, with each domain labelled by the vendor guarding it. Whiskers are 95% confidence intervals.
Raw results of 330.8 thousand data points from the AIMultiple benchmarks this index is built from. Download them as a ZIP file containing 12 CSV files and a README.
Want the granular data behind it? Join Premium
Changelog
16 updates- v1.5
Web Data Scraping Benchmarks: Providers are now ranked only if they were tested in at least 80% of the benchmarks. Version updated to v1.5.
- v1.4
Web Data Scraping Benchmarks: Average metadata fields now count only responses that returned parsed JSON; raw HTML answers no longer count as zero fields. Version updated to v1.4.
- v1.3
Web Data Scraping Benchmarks: The index is now built from the raw results of every published web data benchmark, each weighted equally. Version updated to v1.3.
- v1.2
Web Data Scraping Benchmarks: Each benchmark is now scored on its own, and providers with too few samples are left out. Version updated to v1.2.
- v1.1
5 providers entered the index: Bright Data, Decodo, Nimble, Oxylabs, Zyte.
- v1.1
Web Data Scraping Benchmarks: Providers are now scored per benchmark category and per tool type (scraping API or web unblocker). Version updated to v1.1.
- v1.0
Ethical web data benchmark: Added a Methodology limitations section at the end of the benchmark.
Explore Web Data Scraping Benchmarks
- Social Media ScrapingOpen World EvaluationNov 16
Best Instagram Scrapers to Scrape Instagram Data
We tested six web scraping APIs on 1,500 Instagram profiles, posts and reels, 8,000 requests in total, measuring success rate, completion time and the number of parsed metadata fields for each. See our benchmark methodology for more detail on testing process. Bright Data, Apify and Nimble returned JSON for all three Instagram page types, and…
- Social Media ScrapingBenchmarkNov 16
The Best LinkedIn Scrapers
We benchmarked three scraping APIs with 4,500 LinkedIn requests, sending each provider the same 1,500 URLs: 500 profiles, 500 company pages and 500 posts. For every request we measured success rate, completion time, and the number of parsed metadata fields returned. Read our methodology for details about the LinkedIn benchmark. Bright Data was the only…
- Social Media ScrapingBenchmarkNov 16
Best TikTok Scrapers: Scrape Video Data
A TikTok scraper collects public data from TikTok, including video metadata, profile details, engagement metrics, and comments, without using TikTok’s official API. We benchmarked five web scraping APIs with 5,000 TikTok requests, sending each provider the same 1,000 URLs: 500 profiles and 500 posts. For every request we measured success rate, completion time, and the…
- Social Media ScrapingBenchmarkNov 16
Best Social Media Scrapers: 37K Requests Tested
We sent 30,500 test requests to 6 scraping APIs across 6,500 Facebook, Instagram, LinkedIn, TikTok and X pages to find the most reliable social media scraper. Average metadata fields shows how many structured data fields a provider returns per successful page, such as follower count, post date or caption. JSON providers deliver these fields ready…
- Anti-BlockingBenchmarkNov 13
Best CAPTCHA Solving Services
We compared 10 CAPTCHA-solving services based on which CAPTCHA types each one supports, whether it solves CAPTCHAs via AI or human workers, and how it prices. We also stress-tested the four integrated unblockers (Bright Data, Decodo, Oxylabs, Zyte) against the real-world target, Cloudflare’s Under Attack mode. To see how the integrated unblockers hold up against…
- Web Data ScrapingBenchmarkNov 11
Top 6 Job Posting Scraper APIs Compared
We benchmarked 5 leading web scraping providers across 5 major job platforms by running 12,500 requests in total, then measured each provider’s success rate, completion time, and metadata output. You can read benchmark methodology section for more details on the testing process = supported, returns HTML = supported, returns structured data = no data returned…
- Web Data ScrapingBenchmarkNov 11
Ethical & Compliant Web Data Benchmark
As enterprises scale their web data operations, compliance, data, and risk executives increasingly evaluate the associated ethical, reputational, and legal risks. We benchmarked 5 leading web data collection services across 3 dimensions and tested each service with more than 20 potentially unethical scenarios. Our work helps you assess the ethical standing of your data collection…
- Web Data ScrapingOpen World EvaluationNov 6
Top Firecrawl Alternatives: Features & Pricing
Firecrawl is an AI-native web scraping and crawling API that converts dynamic web pages into LLM-ready Markdown and structured data. We compared its alternatives across benchmarks and features. Vendors marked Yes in the CAPTCHA column (Bright Data, Oxylabs and Spider Cloud) solve challenges that have already appeared. Decodo, Nimble, Zyte and Firecrawl keep challenges from…