Web Scraping API

The same service, differently delivered. Collection design, schema validation, human QA and source maintenance are identical to our file-delivered engagements. What changes is that your application initiates the request.

Most clients use both: API for the application path where request-time data matters, and scheduled files into the warehouse for analytics and full-population work, where pulling everything through an API would be slower and more expensive.

Yes, per request. freshness_max_minutes sets the maximum age you will accept; we serve from cache within that window or fetch live if not. The response reports served_from and age_minutes so you always know what you received.

This is deliberately your decision rather than ours. Cached is fast and cheap, live is current and slow, and the right tradeoff differs by feature within the same product.

You get a successful response with partial set true and an errors array naming each failed identifier, an error code and retry_after_s guidance. Codes distinguish source_unavailable, blocked, not_found and field_unavailable.

We never return a short list silently. An API that returns 470 records for a 500-identifier request with no flag is the most common cause of silent production data gaps, and those gaps correlate with the products you care about most.

Versioned endpoints. /v3/extract keeps its contract; a breaking change becomes /v4. Within a version, changes are additive only — new optional fields may appear, existing fields do not change type, meaning or nullability.

Deprecation carries a dated deprecation object in responses plus direct notice, with an overlap period long enough to migrate calmly. There is a machine-readable schema endpoint per version and a changelog.

Agreed to your load profile during scoping rather than assigned as a fixed tier. We would rather size limits to your actual peak than have you discover a ceiling in production.

Every response carries rate with limit, remaining and reset seconds, mirrored in headers. If your load grows, limits are revised inside the retainer rather than triggering a plan upgrade.

Yes. A sandbox key with representative data is issued before contracting, specifically so you can test error semantics, response shapes and your own client code against the real contract rather than a description of it.

We would rather you find an integration problem in sandbox than after signature. The sandbox is part of the free pilot.

Yes, for change-triggered push. Payloads are signed with replay protection, retries use exponential backoff, and undeliverable events go to a dead letter store you can replay from.

Webhooks suit alerting and event-driven architectures better than polling. If your use case is analytics rather than reaction, scheduled files are usually the better fit and we will say so.

We tell you, and the field is marked unavailable with a reason code rather than returning nulls that look like data. Sources do occasionally stop publishing things.

What we will not do is claim nothing will ever change, or quietly substitute an estimate. If a retailer stops publishing a field, no vendor can supply it, and pretending otherwise creates a worse surprise later.

We quote individually. Drivers are source count, field scope, expected request volume and how much of it needs live rather than cached fetching — live fetching is the main cost variable.

Pricing is a fixed monthly retainer sized to your load profile, not per-request metering. That means a traffic spike does not produce a surprise invoice, and it means we scope your peak honestly upfront. One scoping call, a sandbox key within 24 hours, then a fixed monthly quote. Request a quote.