Web Archive APIs: 4 Top Providers Tested & Compared
A web archive holds pages that were collected before you asked for them, so you can pull them without crawling the sites yourself. What each provider stores, and how much of it they will hand over, varies a lot.
We tested 4 of them with sample queries to see how they work and what they return.
Web archive APIs feature comparison
- Raw HTML available: page markup as the server sent it. Webz.io’s terms mention “the full HTML pages”, but the archive returns a
textfield and grants no rights to the raw files. - Bulk delivery to your own storage: a filtered selection written to a destination you control. Common Crawl’s corpus sits on public S3 for you to download; nothing is sent anywhere. Webz.io delivers through its API only.
- Live scraping from the same vendor: covers the product family, not the archive endpoint alone, so it answers whether one supplier can cover both archived and never-archived pages. Exa is the only one where the fallback sits inside the same call.
Price & gratuito credits
Bright Data charges $0.2 per 1.000 pages for data from the last 24 hours and $1 per 1.000 for anything older. Exa charges $7 per 1.000 searches on top of the page price, so locating 1.000 pages before retrieving them adds roughly $0.70, which makes the two identical $1 figures behave differently in practice.
Free credits cover data delivery: querying is gratuito on all four, so the difference is whether gratuito credits also pay for getting the data out. Bright Data allows 100 archive searches per day at no cost, each returning the matching record count and the delivery price before anything is charged, so a job can be sized for gratuito. Webz.io works the same way, with a gratuito sample and a cost estimate, but a dump requires separately purchased prepaid credits. Common Crawl and Exa let gratuito credits pay for actual retrieval.
Web archiving APIs
Bright Data Web Archive API sells access to pages its infrastructure has already collected through the Unlocker and SERP APIs. Instead of running a crawler, you describe what you want with filters, review a cost estimate, and have the matching pages delivered to your own storage.
Filters
Everything runs through one filters object with 20 fields: time range, domain, URL pattern, category, language, country and collection outcome. Time is mandatory, and category is a closed list of 30 values, so pages are classified on the way in.
Two fields describe how the page was collected rather than what it contains. ip_country filters by the country of the exit IP that collected the page, confirming the archive is a by-product of the proxy network. captcha and robots_block return only records where a CAPTCHA fired or a robots rule applied, so blocked attempts are stored rather than discarded.
Searching and cost
Searching is gratuito, up to 100 queries per day, and the response carries everything you need to size a job before paying: the matching record count, the estimated delivery cost, and a split showing how many pages sit in the 24-hour cache tier versus the archive tier.
- Coverage of sites that block crawlers: Common Crawl holds zero pages for zillow.com across an entire monthly crawl. Bright Data returned 13.282 records for a single 24-hour window.
- Query speed scales with the window: A 240-day query on redfin.com returned 59.831 records in 1 minute 53 seconds. The 24-hour zillow.com query returned in 10.8 seconds.
- Two storage tiers: Data from the last 24 hours costs $0.2 per 1.000 pages. Anything older moves to S3 Glacier Deep Archive at $1 per 1.000. Delivery time follows: our 24-hour job arrived in about 30 seconds, while the 240-day job stayed in
restoring_archive_datafor tens of minutes. - Delivery to your own storage: Amazon S3, Azure Blob Storage, Google Cloud Storage or webhook. We tested the webhook path; ten pages arrived as a single tar with one
.html.gzand one.meta.jsonper page. - Sample before committing:
max_entriescaps how many files a delivery includes, so a 59.831-record query can be sampled with ten pages for a fraction of a cent.
What is actually in the files
Each page arrives as the server sent it, with nine metadata fields: domain, url, timestamp, language, category, ip_country, content_type, charset and status_code. To check whether that raw HTML is usable without a browser, we opened ten archived Zillow pages, stripped scripts and tags, and searched the remaining text for listing fields.
Common Crawl is a nonprofit that publishes a crawl of the open web roughly once a month and gives it away. There is no account, no key and no quota. The trade is that you get files rather than a service: nothing is delivered to you, and anything beyond a single page lookup means running your own processing.
Choosing a format
Each crawl comes in several formats. Raw HTTP responses carry the original HTML. A metadata extract carries links, titles and headers as JSON. A plain-text version carries the article text with markup removed. Two smaller sets cover robots.txt files and non-200 responses.
The choice matters because it is one-way. Plain text is far smaller and quicker to work with, but tags, attributes and embedded JSON-LD are gone from it, so structured extraction has to start from the raw set.
There is no filtered delivery. You pick a format, download what you need, and process it yourself.
Finding a page
Two mechanisms exist and they answer different questions. The CDX server handles single-URL lookups through a web form or an API. The columnar index, published as Parquet, handles bulk questions through SQL.
The CDX interface asks for two things: which monthly crawl to search, from a list running back to 2008, and a URL pattern.
Results come back as JSON, one line per capture. Three fields matter more than the rest: the WARC filename, a byte offset and a record length.
Two things are visible in the output. The same URL appears several times with different timestamps, so a single monthly crawl is not a single snapshot. And failed responses are stored separately, with redirects and errors in a crawldiagnostics folder rather than alongside successful fetches.
Pulling one page without downloading the file it lives in
The filename, offset and length let you request a single record with an HTTP Range header. We tested this on a Redfin listing page.
The CDX lookup returned in under a second and the byte-range fetch took 0.38 seconds for 210 KB. All seven listing fields we searched for, including price, bedrooms and address, were present in the raw HTML with no JavaScript executed.
Bulk filtering with SQL
For anything larger than a single URL, the columnar index is the route. It is Parquet, it is queryable over HTTPS, and it exposes 33 columns covering URL, host, TLD, language, MIME type, fetch status and timestamp.
Queries against a single shard of 7.29 million rows returned in well under a second each, with nothing written to disk. One detail to know before using it: the shards are keyed by a reversed host name, so each file holds a narrow alphabetical slice rather than a random sample. Querying an arbitrary shard for a specific domain will usually return nothing.
How much it holds for a given domain
We queried the index for two domains, redfin.com and zillow.com, locating the shard that covers each one and counting its rows.
redfin.com returned 14.406 pages, all of them HTTP 200. zillow.com returned none. This is not partial coverage or a failed fetch; the domain is not in the index. Common Crawl honours robots.txt, so a site that disallows its crawler is absent rather than thinly represented.
We pulled one archived page from each of eight large sites and measured how much text survived without a browser. Two of them had no usable HTML page in the archive at all. Among the rest, most were server rendered, including Airbnb and Walmart. TripAdvisor was the exception, returning an 18 KB page that held 282 characters of visible text.
Cost and licence
The data is gratuito and the storage bucket is open without an account. What you pay is compute and bandwidth, which is why processing in the same cloud region as the data is recommended.
The licence is not an open data licence. Common Crawl grants a limited licence to the service, states that it cannot license the page content itself, and requires users to indemnify it against claims arising from use of the content “in connection with artificial intelligence, machine learning, or other similar technologies”. Total liability is capped at $100.
Webz.io sells access to an archive of news articles, blog posts, forum threads and reviews that it has been collecting since 2008. What it returns is not the page but a processed record: the article text plus extracted entities, sentiment scores, social engagement counts and site metadata. Anything older than about 30 days sits in the archive product; more recent material is served by the News, Blogs and Forums APIs.
The query builder
Queries combine a Boolean expression over the text with field-level conditions, joined by AND. Each condition is built from a property, an operator and a value.
One detail is worth knowing before you use it: the operator labelled Equals does not require an exact match. We entered redfin against the site title field and it matched a site whose title is “Redfin Real Estate News”.
What comes back
Each result carries the full text alongside entities, sentiment, social, categories, language, domain_rank, country and a syndication object that records whether the same story appeared elsewhere. There is no raw HTML field. The product page describes what you get as “the whole text of every article, post, thread or review”, and delivery is “via API or by running a query yourself”.
- The archive holds publishers, not listing sites. Searching for Redfin returned Redfin’s corporate news blog, hosted on
convesio.cloud, rather than pages from redfin.com. Querying the site domain field directly forredfin.comreturned nothing, whileconvesio.cloudandyahoo.comreturned results, so the empty answer reflects coverage rather than a broken query. - Bulk collection is restricted by contract. The terms of service prohibit a catch-all asterisk filter and vertical filters such as
site_type:newsorlanguage:english, which are exactly the query shapes a corpus buyer would use. The interface itself suggests addingsite_type:newsto narrow a search. - The archive documentation is unwritten. Four pages exist in the docs navigation, covering data dumps, limits, dataset building and the API reference. Each contains a heading and nothing else.
Exa built its own web index and sells search over it. The company reports more than 500 billion URLs. For this comparison the relevant part is not the search endpoint but /contents, which returns the text of a page from Exa’s cache rather than fetching it live.
Cache or live options
A single parameter controls it. maxAgeHours set to 24 means use the cached copy if it is less than a day old, otherwise crawl. Set to 0 it always crawls. Set to -1 it never crawls, returning only what is already stored. The maximum accepted value is 720 hours, so the parameter cannot reference anything older than 30 days.
We ran two requests with the age limit set to six months. The first asked for a Redfin city page.
It returned CRAWL_NOT_FOUND after 7.07 seconds. The second asked for the Redfin homepage.
It returned the full page contents in 0.010 seconds. Ten milliseconds is shorter than a network round trip, so that response came from storage, not from a fetch. The cache is real, and the seven-second failure was Exa looking and not finding.
Two other defaults are visible in the same screen. Content is capped at 20.000 characters, and “main content only” is on, so navigation and boilerplate are removed by Exa’s own logic rather than yours.
Coverage is patchy at the page level
The homepage was cached; the San Francisco city page was not. Searching the domain shows why that is not a simple presence-or-absence question.
The query returned the Redfin homepage, a “Houses For Sale Near Me” page and a Wichita city page, so deep pages do exist in the index. Coverage is scattered across a domain rather than complete or empty.
Limitations
The same screen carries the sentence that separates Exa from the archives in this comparison: “Number of results: 10. Max: 100. Contact us for more results.”
There is no total count. Common Crawl answered that question with a SQL query and Bright Data answered it with a files_count field.
No bulk export, no delivery to your own storage, no firehose. The Enterprise tier mentions custom indexes, which are tailored indexes served through the API rather than data handed over.
Which one fits which job
These four products are not substitutes. The question is not which is best but which one answers the question you have.
- Raw HTML at scale, including sites that block crawlers. Bright Data is the only one here that writes a filtered selection to storage you control, and the only one holding pages from domains that disallow crawlers.
- A broad corpus at no cost, if you have engineering capacity. Common Crawl gives you complete crawls in WARC, enumerable with SQL, and asks for nothing beyond your own compute. The processing is yours to build.
- News, blog and forum text with entities and sentiment already extracted. Webz.io returns processed records rather than pages, which removes the extraction work but also removes the markup.
- Page context at query time, for an agent or a RAG pipeline. Exa returns cached text in milliseconds and falls back to a live fetch when a page is not stored, inside the same call.
Aggiungi come fonte preferitaHow we tested web archive APIs
We worked inside each product rather than from its documentation. Every provider was driven through its own console or playground first, then through its API, so we could see the request that the interface generates and what comes back before anything is charged.
For each one we built a query, ran it, and read the raw response rather than the rendered summary. Where the product returned a count, a cost estimate or a status object, we recorded those fields directly. Where it offered a sample or a delivery, we took it, unpacked the files and opened them.
Small scripts handled the parts that needed repeating or timing: fetching a single record out of a multi-terabyte archive, decompressing a delivered package and searching the raw HTML for specific fields, querying a columnar index with SQL, and measuring how long a cached response takes against one that has to be crawled. Those scripts are linked next to the results they produced.
Cita questa ricerca
Scegli il formato adatto a dove pubblicherai. Incollare la versione con link nel tuo CMS preserva il backlink.
@misc{sipi2026,
author = {Şipi, Nazlı},
title = {{Web Archive APIs: 4 Top Providers Tested & Compared}},
year = {2026},
month = aug,
howpublished = {\url{https://aimultiple.com/web-archive-api}},
note = {AIMultiple. Consultato il 10 Agosto 2026}
}
Sii il primo a commentare
Il tuo indirizzo email non verrà pubblicato. Tutti i campi sono obbligatori. I commenti vengono lasciati nella loro lingua originale.