Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
216.73.216.35
{
  "geoplugin_status":429,
  "geoplugin_message": "Blacklisted due to sending too many requests to geoplugin.net. Consider whitelisting your IP or domain",
  "geoplugin_url": "https://www.geoplugin.com/premium/"

}
http://www.geoplugin.net/php.gp?ip=216.73.216.35
Array
(
    [success] => 
    [message] => You've hit the monthly limit
)
Array
(
    [status] => success
    [country] => United States
    [countryCode] => US
    [region] => OH
    [regionName] => Ohio
    [city] => Columbus
    [zip] => 43215
    [lat] => 39.9625
    [lon] => -83.0061
    [timezone] => America/New_York
    [isp] => Amazon.com
    [org] => Anthropic, PBC
    [as] => AS16509 Amazon.com, Inc.
    [query] => 216.73.216.35
)
How-to-Scrape-E-Commerce-Website-Data-to-Compare-Prices-Using-Python-Part-2

In Part 1 of the two-part series on data scraping e-commerce websites for price comparison, we used the Selenium-Python package to automate the procedure of extracting product prices and names from the Lazada website.

In Part 2 here, we will continue scraping on the Shopee website. Here we will concentrate on particular challenges with extracting the Shopee website rather than repeating the steps in Part I. We will also introduce a substitute to Selenium that worked better!

So, let’s begin!

Scraping the Shopee website

wasn’t easy while using a Selenium tool, and we have highlighted four extra complexities a Shopee website had and a Lazada website hadn’t:

Using-Selenium

Popup Alerts (Extra Complexity = Low)The initial issue we meet is popup alerts, which come when you search:

Popup-Alerts-Extra-Complexity-Low

We can automate clicking away from popup boxes using Selenium with the given script:

We-can-automate

2. Different Prices for the Similar Item (Extra Complexity = Low)

Different-Prices-for-the-Similar-Item-Extra-Complexity-Low

We also get that at times in Shopee search outcomes, one item might have two different pricing figures with a similar class name. Different prices imitate a pricing range where an item has a volume discount:

Using Selenium, we can stipulate the particular figure we need by using an XPath selector to choose the second span component that reflects the initial figure:

3. Search Gives 50 Items Every Page in Search But Merely 15 Got Selected (Extra Complexity = High)

Search-Gives

The Shopee site is a dynamic site, where page elements look dynamically only while scrolling down a page. It isn’t unusual because it helps a page in loading quicker without immediately loading all elements (Facebook works in the similar way).

However, this needs to automate scrolling to bottom of a page like you would do physically, with shorter waiting time for all page fundamentals to appear.

However-this-needs-to-automate

Also, Selenium allows automation to do browser scrolling, however the script for the particular automation could be lengthy because you might need to imitate the manual procedure of scrolling a bit more, and wait a few seconds for page elements to come, rinse and repeat till you reach end of a page.

We could write the script like this:

Here we can see that the code has become much more composite, and the automation procedure has also become slower with extra pause times.

Here-we-can-see

4. The Product Name Elements Just Can’t Be Chosen

As observed earlier, the product names can’t be selected although they could be recognized with either XPath or class selectors and could be seen with a Chrome inspect tool. Due to that, running find_element doesn’t reoccurrence the anticipated item names, only empty strings.

We’ll have to write a few Javascript codes to deploy a CSS property, the language we are extremely unfamiliar with.

Fortunately, we found an easy way of scraping Shopee data: using Shopee’s API to ask for search results.

We were extremely lucky to find that on the web. Not all the websites will get or will share the API with you. Because Shopee helps you use the API to extract product information directly, it becomes much easier to utilize that rather than automating the extraction procedure with Selenium using the given code:

We-were-extremely-lucky

Now, we will make a pandas dataframe for organizing all the data:

Now-we-will-make-a

Printing output data of a dataframe offers the given results:

Printing-output-data Printing-output-data-of-02

With a Lazada dataset, we would also require to conduct cleaning with the dataset. The key things we have to do include:

With-a-Lazada-dataset
  • 1. Transforming the pricing column from integer types into two-decimal float types
  • 2. Removing unrelated entries from a dataset
  • 3. Removing twin packs

Now, it’s time to combine Shopee and Lazada datasets! We do that by utilizing a pandas concatenation technique:

Now we need to compare between these two platforms. We could print a dataframe statistical structures using a describe method:

Now-we-need

We would plot data using similar box plot created in the Part 1:

We-would-plot-data

And that’s it! Depending on one item comparison, it does look that Shopee is the cheaper platform (having extra items).

Some notes before we finish off:

Some-notes-before-we-finish

a) It’s useful to organize a price comparison between various time periods to analyze a pricing trend of any particular item. To do that, we could add a datetime column as well as save this to the csv file.

b) Though you can extract other items just by changing a keyword_search variable, you might have to clean a dataset otherwise from the given example.

c) This example is the small dataset, and so the cleaning and scraping exercise was much quicker.

That’s it for now!

For more information about scraping e-commerce website data to compare prices using Python, contact Actowiz Solutions now!

You can also reach us for all your mobile app scraping and web scraping services requirements.

216.73.216.35
{
  "geoplugin_status":429,
  "geoplugin_message": "Blacklisted due to sending too many requests to geoplugin.net. Consider whitelisting your IP or domain",
  "geoplugin_url": "https://www.geoplugin.com/premium/"

}
http://www.geoplugin.net/php.gp?ip=216.73.216.35
Array
(
    [success] => 
    [message] => You've hit the monthly limit
)
Array
(
    [status] => success
    [country] => United States
    [countryCode] => US
    [region] => OH
    [regionName] => Ohio
    [city] => Columbus
    [zip] => 43215
    [lat] => 39.9625
    [lon] => -83.0061
    [timezone] => America/New_York
    [isp] => Amazon.com
    [org] => Anthropic, PBC
    [as] => AS16509 Amazon.com, Inc.
    [query] => 216.73.216.35
)

Start Your Project

US

Additional Trust Elements

✨ "1000+ Projects Delivered Globally"

⭐ "Rated 4.9/5 on Google & G2"

🔒 "Your data is secure with us. NDA available."

💬 "Average Response Time: Under 12 hours"

From Raw Data to Real-Time Decisions

All in One Pipeline

Scrape Structure Analyze Visualize

Look Back Analyze historical data to discover patterns, anomalies, and shifts in customer behavior.

Find Insights Use AI to connect data points and uncover market changes. Meanwhile.

Move Forward Predict demand, price shifts, and future opportunities across geographies.

Industry:

Coffee / Beverage / D2C

Result

2x Faster

Smarter product targeting

★★★★★

“Actowiz Solutions has been instrumental in optimizing our data scraping processes. Their services have provided us with valuable insights into our customer preferences, helping us stay ahead of the competition.”

Operations Manager, Beanly Coffee

✓ Competitive insights from multiple platforms

Industry:

Real Estate

Result

2x Faster

Real-time RERA insights for 20+ states

★★★★★

“Actowiz Solutions provided exceptional RERA Website Data Scraping Solution Service across PAN India, ensuring we received accurate and up-to-date real estate data for our analysis.”

Data Analyst, Aditya Birla Group

✓ Boosted data acquisition speed by 3×

Industry:

Organic Grocery / FMCG

Result

Improved

competitive benchmarking

★★★★★

“With Actowiz Solutions' data scraping, we’ve gained a clear edge in tracking product availability and pricing across various platforms. Their service has been a key to improving our market intelligence.”

Product Manager, 24Mantra Organic

✓ Real-time SKU-level tracking

Industry:

Quick Commerce

Result

2x Faster

Inventory Decisions

★★★★★

“Actowiz Solutions has greatly helped us monitor product availability from top three Quick Commerce brands. Their real-time data and accurate insights have streamlined our inventory management and decision-making process. Highly recommended!”

Aarav Shah, Senior Data Analyst, Mensa Brands

✓ 28% product availability accuracy

✓ Reduced OOS by 34% in 3 weeks

Industry:

Quick Commerce

Result

3x Faster

improvement in operational efficiency

★★★★★

“Actowiz Solutions' data scraping services have helped streamline our processes and improve our operational efficiency. Their expertise has provided us with actionable data to enhance our market positioning.”

Business Development Lead,Organic Tattva

✓ Weekly competitor pricing feeds

Industry:

Beverage / D2C

Result

Faster

Trend Detection

★★★★★

“The data scraping services offered by Actowiz Solutions have been crucial in refining our strategies. They have significantly improved our ability to analyze and respond to market trends quickly.”

Marketing Director, Sleepyowl Coffee

Boosted marketing responsiveness

Industry:

Quick Commerce

Result

Enhanced

stock tracking across SKUs

★★★★★

“Actowiz Solutions provided accurate Product Availability and Ranking Data Collection from 3 Quick Commerce Applications, improving our product visibility and stock management.”

Growth Analyst, TheBakersDozen.in

✓ Improved rank visibility of top products

Trusted by Industry Leaders Worldwide

Real results from real businesses using Actowiz Solutions

★★★★★
'Great value for the money. The expertise you get vs. what you pay makes this a no brainer"
Thomas Gallao
Thomas Galido
Co-Founder / Head of Product at Upright Data Inc.
Product Image
2 min
★★★★★
“I strongly recommend Actowiz Solutions for their outstanding web scraping services. Their team delivered impeccable results with a nice price, ensuring data on time.”
Thomas Gallao
Iulen Ibanez
CEO / Datacy.es
Product Image
1 min
★★★★★
“Actowiz Solutions offered exceptional support with transparency and guidance throughout. Anna and Saga made the process easy for a non-technical user like me. Great service, fair pricing highly recommended!”
Thomas Gallao
Febbin Chacko
-Fin, Small Business Owner
Product Image
1 min

See Actowiz in Action – Real-Time Scraping Dashboard + Success Insights

Blinkit (Delhi NCR)

In Stock
₹524

Amazon USA

Price Drop + 12 min
in 6 hrs across Lel.6

Appzon AirPdos Pro

Price
Drop −12 thr

Zepto (Mumbai)

Improved inventory
visibility & palniring

Monitor Prices, Availability & Trends -Live Across Regions

Actowiz's real-time scraping dashboard helps you monitor stock levels, delivery times, and price drops across Blinkit, Amazon: Zepto & more.

✔ Scraped Data: Price inights Top-slling SKUs

Our Data Drives Impact - Real Client Stories

Blinkit | India (Relail Partner)

"Actow's helped us reduce out of ststack incidents by 23% within 6 weeks"

✔ Scraped Data, SKU availability, delivery time

US Electronics Seller (Amazon - Walmart)

With hourly price monitoring, we aligned promotions with competitors, drove 17%

✔ Scraped Data, SKU availability, delivery time

Zepto Q Commerce Brand

"Actow's helped us reduce out of ststack incidents by 23% within 6 weeks"

✔ Scraped Data, SKU availability, delivery time

Actowiz Insights Hub

Actionable Blogs, Real Case Studies, and Visual Data Stories -All in One Place

All
Blog
Case Studies
Infographics
Report
July 30, 2025

Why WebMD Drug Information Scraping Is Essential for Extracting Accurate Pharmaceutical Data?

Discover why WebMD Drug Information Scraping is vital for extracting accurate pharmaceutical data, dosage details, side effects, and drug interactions.

thumb

How U.S. Startups Leveraged the Lazada Grocery Dataset for Smarter Delivery Operations & Faster Market Penetration

Discover how U.S. startups used the Lazada grocery dataset to enhance delivery operations and speed up market entry with real-time retail and logistics insights.

thumb

Raksha Bandhan & Independence Day 2025: Travel Price Surge or Discount Season?

Explore how Raksha Bandhan & Independence Day 2025 affect airfare & hotel rates using Actowiz Solutions' travel scraping tools. Data reveals price hikes or discounts.

thumb

Scraping Food Delivery Data for Smart Digital Menu Systems in India

Discover how scraping food delivery data powers Smart Digital Menu Systems in India with real-time pricing, trends, and customer preference insights.

July 30, 2025

Why WebMD Drug Information Scraping Is Essential for Extracting Accurate Pharmaceutical Data?

Discover why WebMD Drug Information Scraping is vital for extracting accurate pharmaceutical data, dosage details, side effects, and drug interactions.

July 30, 2025

Tata CLiQ Personal Care Product Data Scraping - How to Extract Actionable Insights Easily

Tata CLiQ Personal Care Product Data Scraping helps brands extract insights on pricing, reviews & trends to boost product strategies and online visibility.

July 30, 2025

Amazon Seller Competitor Review Analysis - The Secret to Outselling Your Rivals

Boost sales with Amazon Seller Competitor Review Analysis—uncover insights from rival reviews to improve product strategy and outperform competition.

thumb

How U.S. Startups Leveraged the Lazada Grocery Dataset for Smarter Delivery Operations & Faster Market Penetration

Discover how U.S. startups used the Lazada grocery dataset to enhance delivery operations and speed up market entry with real-time retail and logistics insights.

thumb

Raksha Bandhan & Independence Day 2025: Travel Price Surge or Discount Season?

Explore how Raksha Bandhan & Independence Day 2025 affect airfare & hotel rates using Actowiz Solutions' travel scraping tools. Data reveals price hikes or discounts.

thumb

Competitive Benchmarking Using Amazon eCommerce Datasets

Discover how Amazon eCommerce Datasets enable competitive benchmarking, offering deep insights into pricing, trends, and product performance analysis.

thumb

Scraping Food Delivery Data for Smart Digital Menu Systems in India

Discover how scraping food delivery data powers Smart Digital Menu Systems in India with real-time pricing, trends, and customer preference insights.

thumb

99acres and MagicBricks Data Extraction - Real Estate Market Trends in India

Explore MagicBricks data extraction and 99acres insights to analyze real estate market trends in India, from pricing shifts to demand patterns across cities.

thumb

Real-Time Used Car Dataset from Carfax for Accident-Vehicle Insights

Explore how a Real-Time Used Car Dataset from Carfax enables accident-vehicle tracking, helping dealers, insurers, and buyers make informed, data-driven decisions.