Shopify Store Data Scraping

Store discovery is the actual project

On Amazon you know where the catalogue is. On Shopify there is no catalogue — there are hundreds of thousands of independent stores, and the ones you care about are a subset you have to define and find.

How discovery works

• Seed list from you — competitor and category brands you already know. Always the starting point.
• Platform fingerprinting to confirm a domain is Shopify-hosted before we build against it.
• Category expansion from public directories, category pages and brand listings.
• Continuous new-store detection in your category, which for DTC-heavy categories is a genuine competitive signal.

We deliver the discovered store list with discovery_method and platform_confirmed on each, so you can see how each store entered the set. And we scope discovery separately from collection, because a discovery-only engagement is sometimes the whole requirement — knowing who is competing in your category is often the question.

Variant structure carries the pricing

Shopify's data model is product-with-variants, and on DTC stores the variant is where price, availability and the actual purchasable unit live. Product-level collection loses most of what matters.

• Price varies by variant — size, colour, subscription versus one-time, bundle tiers.
• Availability is per variant, so a product showing available may have most variants sold out.
• Subscription pricing is common on DTC and is a different commercial model from one-time purchase.
• Bundle and multipack variants change unit economics and need unit-price normalisation.

We collect at variant level with variant_available, subscription pricing captured as its own field where offered, and unit price computed on a consistent basis. The same reasoning as size curves in fashion and shade curves in beauty: the purchasable unit is the record.

DTC velocity is much higher than retail

Independent brands launch, reprice and delist faster than retailers do. On a Shopify store set, the interesting signal is frequently movement rather than state.

• Launch velocity — new products per store per month, which indicates how actively a brand is building range.
• Price change frequency, which on DTC is often testing rather than promotion.
• Delisting rate, showing what did not work.
• New store entry in your category, which retail data cannot show at all.

All of this needs continuous collection with first-seen dates. A monthly snapshot of a DTC store set tells you what exists and almost nothing about what is happening.

Where publicly detectable, we also capture theme and app indicators. That is a genuine signal for anyone selling to DTC brands, but we treat it as observation rather than as a verified stack: a detectable indicator means a signature was present, not that a tool is definitely in use.