How an AI Commerce Platform Built a Complete Luxury Retail Catalogue — Every Category and Attribute

What we collected

How we built it

Source analysis before any collection

We began with a structural survey of the site rather than a scrape. The goal was to answer four questions: how is the taxonomy expressed, how are products addressed, how are variants represented, and where does each attribute actually live. That survey produced a written source map — a category-by-category inventory of which fields are present, which are optional, where the same concept appears under different labels, and which categories carry attributes no other category has. Every later decision in the pipeline traces back to that map.

Taxonomy-first crawl design

Because completeness was the stated requirement, we did not discover products by following links opportunistically. We first enumerated the full category tree to leaf level, then treated each leaf as a bounded collection unit with its own expected product count. This matters for a simple reason: you cannot prove a catalogue is complete without a denominator. With leaf-level enumeration, each unit has a target count from the site's own listing metadata, and any unit that comes back short is flagged rather than silently accepted.

Variant expansion

We modelled the catalogue at two levels: a product entity and a variant entity, joined by product_id. Shared attributes — brand, description, category, composition — live on the product. Everything that can differ by combination — price, availability, size label, colour, SKU — lives on the variant. Variant axes were read per category from the source map, not guessed. Where a combination was not purchasable, we recorded it as unavailable rather than omitting it, so the client can tell "does not exist" apart from "temporarily out of stock".

Attribute extraction and per-category schemas

Each category got an extraction profile: a set of rules mapping source components to target fields, with a fallback path for optional blocks. Free-text attribute blocks were parsed into key/value pairs where the source used a consistent label pattern, and retained verbatim in attributes_extra where it did not. Nothing was inferred. If a value was not stated on the page, the field is null — never filled with a plausible guess.

Reliability and responsible collection

A full-catalogue pass against a large site is a sustained workload, not a burst. We ran it as a paced, distributed job designed to behave like ordinary traffic: conservative per-source concurrency, randomised request ordering, session and header hygiene consistent with real clients, and a rendering path only for components that genuinely require it. Rate limiting was applied on our side as a hard ceiling rather than a target. Reliability came from checkpointing. Every leaf unit and every page within it was tracked in a job ledger, so a failed unit retried in isolation with exponential backoff rather than restarting the pass.

Normalisation and matching

Brands were resolved against a controlled list, with alias handling for punctuation and legal-entity variants. Colour names were mapped to a colour family vocabulary while retaining the retailer's own label. Size labels were parsed into a normalised value plus a declared size system. Prices were stored as decimals with explicit currency, never as formatted strings. Product identity was anchored on the retailer's own identifier where exposed, with a fallback composite key, so a re-run reconciles against the previous load instead of creating duplicates.

Quality assurance

• Coverage reconciliation. Per-leaf captured count versus the site's declared count, with a tolerance threshold that triggers investigation rather than a pass.
• Taxonomy integrity. Every product resolves to a leaf; no orphan nodes; no product attached to an interior node only.
• Variant completeness. Every product has at least one variant; variant axes match the category's expected axis set.
• Attribute fill-rate monitoring. Per field, per category, against the previous load; a sudden drop is treated as an extraction fault, not a catalogue change.
• Image validation. Every URL resolves, primary image present, no duplicate primaries within a variant.
• Price sanity. Currency present, non-negative, price_current ≤ price_original where both exist, outliers flagged by category.
• Text integrity. Encoding checks for accented characters and designer names, no truncated descriptions, no HTML leakage.
• Duplicate detection. Exact and near-duplicate checks on product_id, URL and name-plus-brand.
• Manual spot review. Sampled rows checked against the live page each load, across every category.

Why this approach worked

The decision that carried this project was mapping the taxonomy before collecting anything. It cost time up front and it made every downstream guarantee possible: a per-leaf denominator to measure completeness against, a per-category schema that respects how different merchandise actually behaves, and a variant model that a recommendation engine can use without deduplicating first.

The second decision was refusing to infer. An AI commerce platform is trained on what you give it, so a plausible-looking guessed attribute is worse than a null — it is wrong in a way nobody can detect downstream. Leaving fields empty when the source is silent produced a smaller dataset than a maximalist approach would have, and a far more trustworthy one.