We scraped 10,000 live domains and 100 marketplaces using products from six web data infrastructure companies.We benchmarked these tools to see how well they handle enterprise web data use cases.
Web unblocking benchmark results
We benchmarked 4 leading web data providers across the top 10,000 domains, running a total of 260,000 requests. Each provider was tested at multiple concurrency levels.
Web unblocking benchmark markdown output performance
We also tested the markdown output of these web data providers.
For the detailed methodology and analysis, read our web unblockers benchmark.
E-commerce scraping benchmark
Because e-commerce domains carry the heaviest anti-bot protection, we benchmarked the web data scraping providers on the top 100 e commerce domains.
See our e-commerce scraping benchmark for details.
Web data providers pricing
Costs are based on our spend in the e-commerce and web unblocker benchmarks, scaled to each volume tier.
Web data scraping benchmark results by use cases
We benchmarked each category with its own methodology and target domains. For the full methodology and domain-level results, see:
Marketplace scraping benchmark
Success rates hit nearly 100% for every provider in some benchmarks. We used available metadata fields and completion time instead. Most of our web data scraping articles also break down the metadata fields each provider returns per domain.
This metric applies only to products with a dedicated API that returns JSON. When a provider returns raw HTML, we extract the fields ourselves using CSS selectors and record 0 metadata fields for it. A 0 therefore means the provider returned HTML rather than structured data, not that the request failed.
See our App Store scraping and Google Play scraping benchmarks for the full breakdown.
Video scraping benchmark
Read video scraping benchmark for more details about our methodology.
Learnings from web data scraping benchmarks
Since the legality of collecting web data continues to be challenged, many businesses do not yet have a web data strategy and may not be aware of all solutions. Enterprises that need to collect web data typically value receiving structured, high-quality data with minimal technical effort via cost-effective, reliable services.
To achieve the goals above, enterprises need to:
- Outline the types of pages that they need to crawl
- Leverage web scraping APIs when they are available, since they minimize tech effort on the client side by providing structured data, and they are cost-effective. They cost about the same as residential proxies, even though residential proxies provide unstructured data.
Before dedicated scraper APIs, we relied on unblockers for our own company’s data collection needs. A single endpoint covered every site: we fetched the HTML and parsed the fields out of it with CSS selectors. Our tech team was burdened every time our target websites changed their design. After realizing the scope of web scraping APIs and seeing that they are not more expensive than unblockers, we switched mostly to using scraping APIs in our data collection workflows. Scraper APIs return JSON, they are built for the specific sites they target, they keep working when the site changes, and they come with metadata fields.
For the remaining pages, we rely on:
- Web unblockers for hard-to-scrape pages, as they are the solution that consistently returns successful results over 90% of the time without complex configuration.
- Datacenter or residential proxies for other pages if the enterprise’s tech team is comfortable with configuring proxies and maintaining these configurations to ensure high success rates.
- Mobile proxies for mobile responses, plus other proxies for more niche use cases.
Compare web data providers’ performance, price & reliability
In web-scraping APIs, you can choose:
- Bright Data for its market-leading range of web scraping APIs at cost-effective prices with detailed results. Many Bright Data SERP and e-commerce APIs return more data points than those of competitors.
- Apify for its market-leading range of web scraping APIs thanks to its community-driven scraper approach. However it was the most expensive provider in our benchmarks.
- Others opportunistically (e.g. Decodo returned the most data points for Instagram posts).
In unblockers, leading products include:
- Bright Data is more successful than most in real-world tests and significantly more successful in more difficult scenarios, such as scraping websites that regularly present JavaScript challenges. Bright Data also has fastest unblocker. It provides the second-lowest-priced unblocker in unblocker benchmark.
- Nimble has the lowest-priced unblocker in real-world tests.
Learn more about web unblockers and see detailed results.
Proxies: You can rely on any of the providers based on your technical team’s preferences and pricing. This is because results vary significantly based on:
- Time: While publishers improve their anti-scraping measures, web data infrastructure providers continually receive new IPs and refine their approaches. The success rate depends on the test time.
- Request: Success of a request via a proxy depends on how the request is sent. For example, user-agent choice or the delay between requests significantly impacts the success rate.
How to choose the right data collection solution
1. Enterprise web data requirements:
Enterprises include diverse businesses. For example, businesses with e-commerce operations and hedge funds require high volumes of data to feed their models (e.g. dynamic pricing, stock replenishment). Their requirements include:
- Buyer-related dimensions
- High volume
- Batch
- Price & quality sensitivity
- Want to receive structured data
- Website-related dimensions
- Easy & difficult-to-crawl
- Static and dynamic
- Mixed
To achieve these requirements, enterprises need:
- Capabilities to support their requirements:
- A wide selection of web scraping APIs that return detailed results with a high success rate to deliver structured data and satisfy their quality sensitivity. Measurement: Share of types of web pages to be crawled for which a web scraping API is provided. This would depend on the types of pages that each enterprise targets.
- An unblocker for difficult-to-crawl websites. Measurement: Crawler’s success rate for a wide range of web pages, including the most challenging ones.
- Unblocker integration with browsers to enable interacting with websites for dynamic scraping. Measurement would include checking the availability or lack of this browser.
- Cost-effective services to satisfy their price sensitivity. For measurement, the price to crawl a set of web pages is measured.
- Reliability:
- A resilient web data infrastructure to handle high-volume batch queries. Measurement is based on how the success rate degrades during load testing. Most resilient networks should not experience drastic declines in success rates when answering tens of thousands of parallel queries.
2. Web data requirements for small, highly technical teams:
If your data collection costs will determine your company’s profitability, and if you are a highly technical team, we recommend relying on proxies to reduce costs.
Finally, all buyers should pay attention to proxy pricing; therefore, we calculated prices for the same packages for all major web infrastructure providers:
See proxy pricing methodology for details.
Add as preferred sourceWeb scraping industry updates
AI crawler restrictions are becoming a core challenge for scraping. Large infrastructure providers and publishers are shifting from passive robots.txt guidance to active control.
For example, Cloudflare offers AI crawler blocking and AI Labyrinth, which uses hidden nofollow links to trap crawlers that ignore no-crawl rules. These changes make crawler identity, robots.txt compliance, permission handling, and data provenance more important for scraping teams.
Recent litigation shows that scraping risk is expanding beyond the question of whether public data can be accessed. Platforms are increasingly using contract, unfair competition, trespass, copyright, and anti-circumvention theories against large-scale scraping, especially when the data is used for AI products or resold as a service. 1
Web scraping for machine learning (ML)
Scrapers are now LLM-native. Tools such as Firecrawl, Crawlbase and, Bright Data offer features that automatically convert raw HTML into Markdown or clean JSON, specifically formatted for Retrieval-Augmented Generation (RAG) applications.
Web Scraping vs. Screen Scraping
Web scraping targets underlying data structures such as the DOM, APIs, and JSON. Screen scraping is now a specialized tool for legacy system recovery, capturing the visual user interface as pixels and text via OCR, and is mainly used for desktop applications.
Dimensions of web data requirements
We are not covering every type of web data use case here. Many web data users have multiple one-off requests. That is not the focus of this report.
We have seen that enterprises typically have recurring web data needs to monitor sentiment, prices, or other rapidly changing metrics. Therefore, we have focused on companies that continuously use web data. These dimensions are:
Buyer-related dimension
1. Volume:
- High volume, meaning 100 GB/month or more
- Low volume for any lower volume
2. Time sensitivity:
- Real-time: When web data, in raw or processed form, is served to human end users while they use applications, real-time responses are essential.
- Batch: Response times are not critical as long as results are received within tens of seconds. In most use cases, businesses batch process incoming web data to update their systems.
3. Quality sensitivity:
- Quality-sensitive: All web data solutions sometimes return empty responses when blocked by websites. Companies that want to spend limited time resending requests prefer solutions with higher success rates.
- Price-sensitive: Given that their other requirements are satisfied, these businesses want the lowest price and are willing to run their data collection systems multiple times to achieve higher-quality results.
- Price & quality sensitive: Businesses that want the optimal combination of high success rates and price.
Website-related dimension:
4. Difficulty:
- Difficult-to-crawl websites like Amazon employ numerous anti-scraping technologies.
- Easy-to-crawl websites can be crawled with proxies
- Easy & difficult-to-crawl websites
5. Interactivity:
- Static websites make up most of the web and deliver data via changes in the URL.
- Dynamic websites require users to use a mouse or keyboard to disclose additional information.
- Static and dynamic websites
Methodology
This web data benchmark includes the benchmarks below, and the methodology for each benchmark is explained in its specific page:
- eCommerce scrapers
- Real estate scraping
- Job postings scraping
- Food delivery scraping
- Social media scraping
- Social media scrapers
- Web unblockers
- App Store scraping
- Google Play
- Video scraping benchmark
Proxy pricing methodology
Almost all prices are based on publicly disclosed packages.
However, not all vendors disclose pricing at the same levels. While one vendor may provide pricing for 100 GB of residential proxy usage, another may offer pricing for 50 GB. In cases where their pricing was not public, if vendors share private pricing information with us, we include it in the benchmark, provided it does not change the ranking of vendors.
Our rationale is that we want to share:
- The most accurate pricing possible with our readers
- Pricing levels that are in line with the publicly available prices, which can be constantly monitored.
Unit conversions
For the same product, vendors may provide pricing in GB or in requests; we needed to convert these values between them.
We assume an average page size of ~400KB, based on our measurement of 1,700 e-commerce URLs. Therefore, we thought 1GB would equal 2.5k requests.
Packages
We looked into two packages: the enterprise PoC package and the enterprise package. The Enterprise PoC package is designed to be broadly representative of an enterprise PoC scope:
- 100 GB residential proxies
- 100 GB mobile proxies
- 500 GB datacenter proxies
- 500k unblocker requests
- 500k scraping API requests to Amazon product pages
The enterprise package is the highest-volume package with public pricing. In each product category, we identified the highest volumes offered by each provider and took the highest volume as the volume in the enterprise package for that product:
- 1,000 GB residential proxies
- 1,000 GB mobile proxies
- 5,000 GB datacenter proxies
- 2.5M unblocker requests
- 2.5M scraping API requests to Amazon product pages
Limitations
When enterprises procure such services at high volumes, they are likely to get discounts. Such enterprise discounts are not public and are not included in the benchmark.
Vendor-specific assumptions
Some vendors’ pricing is complex, which requires certain assumptions:
- Apify:
- For datacenter proxies, we assumed that the user buys a $499/month package and pays $0.25/GB for platform usage.
- For scrapers: We took the average price of these two scrapers: junglee~amazon-crawler and tri_angle~walmart-product-detail-scraper
- Oxylabs prices its unblocker on a GB basis. Therefore, we converted its pricing to a per-request model, assuming an average page size of ~400 KB.
- Zyte: The 4th pricing tier was recommended for the websites in our benchmark. We leveraged the HTTP response service.
Limitations and next steps
AIMultiple’s experience may differ from an average user’s experience in these cases: Users can
- Receive faster responses due to caching. Our work aimed to bypass caching in all providers to provide a level playing field.
- Receive fewer successful responses when extracting data from less popular websites since their requests may be blocked due to website health issues.
- Make configuration mistakes, miss KYC requirements, or get blocked when they initially send a high volume of requests. All of these can undermine their experience and success rates. Support teams can swiftly resolve all of these issues.
Finally, network quality will fluctuate, and this benchmark is a series of snapshots taken during a month. It should be representative for that month, but network quality can change after the benchmark.
Acknowledgements & disclaimers for transparency
Providers contributed to our benchmark by providing part or all of the credits used. We thank them for their support of our research.
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{dilmegani2026,
author = {Dilmegani, Cem},
title = {{Web Scraping Roadmap}},
year = {2026},
month = aug,
howpublished = {\url{https://aimultiple.com/web-scraping}},
note = {AIMultiple. Retrieved August 26, 2026}
}Results and timestamps of 0 data points. Download the data used in this article as a ZIP file containing 0 CSV files.
Changelog
9 updates- 2026
Replaced the industry updates section's real browser execution content with AI crawler restrictions and scraping litigation.
Added "Web scraping industry updates in 2026" to the "See pricing methodology for details" section.
- 2025
Added the "Web data collection benchmark results" section, including a detailed analysis of leading products based on performance metrics.
Added a section comparing web scraping with API and screen scraping.
- 2023
Added Real Estate to the applications section.
Updated the list of related articles in the "For more on web scraping" section.
- 2022
Added a data-driven web scraping vendor evaluation to the introduction.
Expanded the "Top 10 web scraping applications/use cases" section.
Added Bright Data to the 'How does it work?' section.
Reference Links
Cem's work at AIMultiple has been cited by leading global publications including Business Insider, Forbes, Morning Brew, and Washington Post, global firms like Deloitte and HPE, NGOs like World Economic Forum, and supranational organizations like European Commission. [1], [2], [3], [4], [5]
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.