Price Comparison Engine Data Scraping

The price is a feed snapshot, and that is not a technicality

Merchants submit product feeds to comparison engines. The engine displays what it last received. The merchant's site may have changed since, and frequently has.

• A price on a comparison engine is a statement about what a merchant submitted, at some point, in some condition.
• It is not evidence of the merchant's current price. On fast-moving categories the gap can be substantial.
• Availability is worse. Feed-based stock status lags reality more than price does, because merchants update price more often than they update availability.

Every row carries source_type as aggregator and price_provenance as feed-derived, plus any freshness indicator the engine publishes. A dataset that presents these prices in the same column as directly-observed retailer prices is mixing two different kinds of claim, and nobody downstream can separate them.

What comparison engines are genuinely good for

Being second-hand does not make the data useless. It makes it good at different things.

• Merchant discovery. Finding out who sells a product at all, including merchants you had not heard of, is what an aggregator is structurally best at.
• Breadth in one pass. One collection reaches many merchants for a product, which is expensive to assemble merchant by merchant.
• Detecting that something changed. A feed price moving is a signal worth following up, even if it is not proof.
• Unauthorised seller leads. A merchant listing your product who is not in your distribution list is a lead worth verifying.

Note the shape of all four: discovery and signal, not measurement. That is the honest use of this source, and scoping a project around it accordingly is the difference between a dataset that works and one that quietly misleads.

Where price truth matters, verify at the merchant

If a client needs the actual price a shopper would pay, the comparison engine is a starting point and not the answer.

The pattern that works: use the aggregator for discovery and breadth, then verify the prices that matter by collecting from the merchant's own site directly. We link the two with verified_at_merchant and merchant_observed_price, so a row carries both the feed figure and the directly observed one where verification was run.

The gap between those two columns, measured across a category, is itself a useful finding — it tells a client how much they can trust aggregator pricing in their market before deciding how much verification to pay for.

Sponsored and ranked placement is not a price signal

Comparison engines rank and promote listings, and paid placement is common. The order in which merchants appear reflects commercial arrangements as much as price.

We capture is_sponsored and result_position where determinable, and we do not treat position as a proxy for competitiveness. A merchant appearing first is not evidence of being cheapest.

For clients using comparison engines to understand their own visibility, position and sponsored status are the fields that matter, and they are captured for that purpose rather than as a pricing input.

Merchant identity resolution across engines is the hard part

The same merchant appears under different display names across engines, and sometimes under several within one engine. A merchant count without a resolution step counts names.

We resolve merchant identity with the rule published, retaining the raw display name, and report both the raw count and the resolved count. Where a merchant cannot be resolved with confidence, that row is marked rather than force-matched.

For brand protection this matters most: a resolved merchant list is actionable, a list of display-name variants is a research task handed back to the client.