We benchmarked 5 leading web scraping providers across 5 major job platforms by running 12,500 requests in total, then measured each provider’s success rate, completion time, and metadata output.
Job posting scrapers benchmark
You can read benchmark methodology section for more details on the testing process
Domain coverage by provider
✅ = supported, returns HTML
✅ ✅ = supported, returns structured data
❌ = no data returned
Job scraping performance by domain
Available metadata fields for job posting APIs
Bright Data is the only provider returning structured JSON for job postings. The table below groups Bright Data’s structured fields into shared categories so you can compare what’s available per platform.
Job scraping benchmark results
Bright Data led the benchmark with a 90% average success rate across the five job platforms. Its setup is split into two integration modes:
- Dedicated Dataset APIs (structured JSON) for LinkedIn, Indeed, and Glassdoor
- Web Unblocker proxy (rendered HTML) for Craigslist and ZipRecruiter
Four domains came in at 100% success rate: LinkedIn, Indeed, Craigslist, and Glassdoor. Completion times depended on the integration. Web Unblocker requests on Craigslist returned in about 1 second on average, LinkedIn in 7, and Indeed in 17. Glassdoor took 53 seconds. ZipRecruiter was the only domain below threshold at 53%, where the Web Unblocker hit token-expired redirects on a portion of the URLs.
Get 25% off Bright Data Web Scraping APIs, promo code API25
Visit WebsiteOxylabs reached a 77% average success rate across the five platforms. The benchmark ran through its Web Scraper API using source: universal, which returns rendered HTML for local parsing.
Four domains performed well: 100% on Craigslist, 100% on Indeed, 98% on LinkedIn, and 90% on ZipRecruiter. Glassdoor was the exception, with most requests timing out at HTTP 408 because the realtime endpoint could not render Glassdoor’s JavaScript-heavy pages within its internal limit. Completion times on the working domains stayed between 11 and 28 seconds.
Get 2,000 free scraping credits
Visit WebsiteDecodo‘s overall performance was the same as Oxylabs, with a 77% average success rate. Its Web Scraper API ran with headless: html and proxy_pool: premium, returning rendered HTML which we parsed locally via CSS selectors.
Per-platform results almost mirrored Oxylabs: 100% on Craigslist, 100% on Indeed, 98% on LinkedIn, 89% on ZipRecruiter, and 0% on Glassdoor. The Glassdoor failure was different though, with most requests rejected at the API level before the page loaded. Completion times on the working domains ranged from 12 to 29 seconds, placing Decodo in the slower half of the field.
Apply SCRAPE30 for 30% offNimble’s overall result was 69%, with most of the loss tied to a single platform. Its Web Extract API ran with browser rendering enabled (render: true, driver: vx10).
Craigslist returned 100%, LinkedIn 86%, Glassdoor 79%, and ZipRecruiter 69%. Indeed dropped to 14% because the rendered pages rarely contained the job-detail DOM elements our selectors targeted. The notable strength here was speed: Indeed, Craigslist, LinkedIn, and ZipRecruiter all returned in 6 to 8 seconds, while Glassdoor was the only outlier at 30 seconds.
Zyte posted the lowest overall success rate at 58%. Its Extract API ran with browserHtml: true, rendering pages through a headless browser. Three domains performed cleanly: 100% on Craigslist, 100% on Glassdoor, and 89% on ZipRecruiter. The other two failed completely:
- LinkedIn returned HTTP 451 Unavailable For Legal Reasons on all 500 requests
- Indeed’s rendered HTML never contained the job-detail DOM elements
Completion times on the working domains ran from 7 seconds on ZipRecruiter to 17 on Craigslist, with Glassdoor at 16.
Job scraping benchmark methodology
We benchmarked 5 leading web scraping providers across 5 major job platforms (LinkedIn, Indeed, Glassdoor, Craigslist, and ZipRecruiter), running 12,500 requests in total. Each provider received the same set of 500 individual job posting URLs per platform, submitted sequentially with a 2-second delay between requests.
Providers and integration
Every provider ran on its own production endpoint, with no custom proxies or third-party middleware in front of them.
Bright Data combined two integration modes. For LinkedIn, Indeed, and Glassdoor it used dedicated Dataset APIs, which return structured JSON. For Craigslist and ZipRecruiter it used the Web Unblocker proxy, which returns rendered HTML.
Oxylabs ran through its Web Scraper API with source: universal, returning rendered HTML on every domain.
Decodo ran through its Web Scraper API with headless: html and proxy_pool: premium, also returning rendered HTML.
Nimble ran through its Web Extract API with render: true and driver: vx10, producing rendered HTML.
Zyte ran through its Extract API with browserHtml: true, again producing rendered HTML.
For HTML responses we parsed the page locally with CSS selectors targeting each platform’s job-detail elements (job title, company name, location, salary, employment type, and a page indicator).
Timeout and rate limiting
Async requests had a 10-minute ceiling on execution. HTTP 429 responses triggered a 30-second backoff with up to 3 retries; anything past that was logged as a failure for the URL.
Validation rules
Each request went through three checks.
The submission check required an HTTP status of 200 to 399 or 404 from the provider. The execution check required async jobs to finish within timeout without errors; sync providers auto-passed. The validation check required at least one of job_title or company_name to be returned as a non-empty string. For JSON providers, this came from the parsed response; for HTML providers, it came from CSS selector matches.
A request that detected a 404 page (HTTP 404, “page not found” content, or a provider’s explicit “dead page” signal) was also counted as valid, since the provider had correctly identified an unavailable listing.
Empty responses with no error were initially counted as valid, then re-checked: if any other provider extracted real job data on the same URL, the empty response was flipped to invalid. 404 detections were exempt from this flip; a provider’s explicit “page doesn’t exist” signal was trusted unless contradicted by real extracted data from another provider.
A run was counted as overall successful only if submission, execution, and validation all passed.
Metrics measured
Validation success rate is the share of URLs that passed all three checks.
End-to-end completion time is wall-clock time from sending the request to getting a response, in seconds. For async providers, this includes polling time until the dataset job finished.
Available metadata fields, for providers returning structured JSON, is the unique field count across all responses computed as a set union. For HTML providers, this is the fixed five-selector CSS schema we used per platform.
FAQs about job scraping
Scraped job data is commonly used for hiring market analysis, salary benchmarking, competitive intelligence on which companies are hiring for which roles, talent pool mapping, recruitment automation, and feeding job aggregators. Companies also use it to track posting volume trends, geographic concentration, and how quickly competitors fill roles.
It depends on the use case. For real-time recruitment automation, daily or hourly scrapes are common. For market reports, weekly or monthly scrapes are usually enough. Job postings tend to be removed quickly once filled, so older data loses value fast.
Scraping publicly accessible data is generally legal in most jurisdictions, but most major job platforms (LinkedIn, Glassdoor, Indeed) have Terms of Service that prohibit automated access. Several have brought legal cases against scrapers in the past. Commercial use cases warrant a legal review, especially when personal data is involved.
Job platforms invest heavily in anti-scraping measures. CAPTCHAs, login overlays, JavaScript-rendered content, frequent layout changes, and IP-based rate limiting are standard. Some platforms also serve different DOM structures to bots versus regular users. These defenses are why many teams rely on managed scraping APIs rather than building their own scrapers.
Be the first to comment
Your email address will not be published. All fields are required.