Start Your Project with Us

Whatever your project size is, we will handle it well with all the standards fulfilled! We are here to give 100% satisfaction.

  • Any feature, you ask, we develop
  • 24x7 support worldwide
  • Real-time performance dashboard
  • Complete transparency
  • Dedicated account manager
  • Customized solutions to fulfill data scraping goals
Careers

For job seekers, please visit our Career Page or send your resume to hr@actowizsolutions.com

How-to-Scrape-E-Commerce-Website-Data-to-Compare-Prices-Using-Python-Part-2

In Part 1 of the two-part series on data scraping e-commerce websites for price comparison, we used the Selenium-Python package to automate the procedure of extracting product prices and names from the Lazada website.

In Part 2 here, we will continue scraping on the Shopee website. Here we will concentrate on particular challenges with extracting the Shopee website rather than repeating the steps in Part I. We will also introduce a substitute to Selenium that worked better!

So, let’s begin!

Scraping the Shopee website

wasn’t easy while using a Selenium tool, and we have highlighted four extra complexities a Shopee website had and a Lazada website hadn’t:

Using-Selenium

Popup Alerts (Extra Complexity = Low)The initial issue we meet is popup alerts, which come when you search:

Popup-Alerts-Extra-Complexity-Low

We can automate clicking away from popup boxes using Selenium with the given script:

We-can-automate

2. Different Prices for the Similar Item (Extra Complexity = Low)

Different-Prices-for-the-Similar-Item-Extra-Complexity-Low

We also get that at times in Shopee search outcomes, one item might have two different pricing figures with a similar class name. Different prices imitate a pricing range where an item has a volume discount:

Using Selenium, we can stipulate the particular figure we need by using an XPath selector to choose the second span component that reflects the initial figure:

3. Search Gives 50 Items Every Page in Search But Merely 15 Got Selected (Extra Complexity = High)

Search-Gives

The Shopee site is a dynamic site, where page elements look dynamically only while scrolling down a page. It isn’t unusual because it helps a page in loading quicker without immediately loading all elements (Facebook works in the similar way).

However, this needs to automate scrolling to bottom of a page like you would do physically, with shorter waiting time for all page fundamentals to appear.

However-this-needs-to-automate

Also, Selenium allows automation to do browser scrolling, however the script for the particular automation could be lengthy because you might need to imitate the manual procedure of scrolling a bit more, and wait a few seconds for page elements to come, rinse and repeat till you reach end of a page.

We could write the script like this:

Here we can see that the code has become much more composite, and the automation procedure has also become slower with extra pause times.

Here-we-can-see

4. The Product Name Elements Just Can’t Be Chosen

As observed earlier, the product names can’t be selected although they could be recognized with either XPath or class selectors and could be seen with a Chrome inspect tool. Due to that, running find_element doesn’t reoccurrence the anticipated item names, only empty strings.

We’ll have to write a few Javascript codes to deploy a CSS property, the language we are extremely unfamiliar with.

Fortunately, we found an easy way of scraping Shopee data: using Shopee’s API to ask for search results.

We were extremely lucky to find that on the web. Not all the websites will get or will share the API with you. Because Shopee helps you use the API to extract product information directly, it becomes much easier to utilize that rather than automating the extraction procedure with Selenium using the given code:

We-were-extremely-lucky

Now, we will make a pandas dataframe for organizing all the data:

Now-we-will-make-a

Printing output data of a dataframe offers the given results:

Printing-output-data Printing-output-data-of-02

With a Lazada dataset, we would also require to conduct cleaning with the dataset. The key things we have to do include:

With-a-Lazada-dataset
  • 1. Transforming the pricing column from integer types into two-decimal float types
  • 2. Removing unrelated entries from a dataset
  • 3. Removing twin packs

Now, it’s time to combine Shopee and Lazada datasets! We do that by utilizing a pandas concatenation technique:

Now we need to compare between these two platforms. We could print a dataframe statistical structures using a describe method:

Now-we-need

We would plot data using similar box plot created in the Part 1:

We-would-plot-data

And that’s it! Depending on one item comparison, it does look that Shopee is the cheaper platform (having extra items).

Some notes before we finish off:

Some-notes-before-we-finish

a) It’s useful to organize a price comparison between various time periods to analyze a pricing trend of any particular item. To do that, we could add a datetime column as well as save this to the csv file.

b) Though you can extract other items just by changing a keyword_search variable, you might have to clean a dataset otherwise from the given example.

c) This example is the small dataset, and so the cleaning and scraping exercise was much quicker.

That’s it for now!

For more information about scraping e-commerce website data to compare prices using Python, contact Actowiz Solutions now!

You can also reach us for all your mobile app scraping and web scraping services requirements.

RECENT BLOGS

View More

How to Use Web Scraping for Inventory Data and Pricing Data on DigiKey?

Learn how to use web scraping for inventory data and pricing data on DigiKey. Follow this guide for step-by-step instructions to automate data extraction efficiently.

How to Scrape Coupon Details from a McDonald’s Store using Python and LXML?

Learn to scrape McDonald’s coupon details using Python and LXML.Follow this guide for step-by-step instructions on automating data extraction efficiently.

RESEARCH AND REPORTS

View More

Analyzing Women's Fashion Trends and Pricing Strategies Through Web Scraping Gucci Data

This report explores women's fashion trends and pricing strategies in luxury clothing by analyzing data extracted from Gucci's website.

Mastering Web Scraping Zomato Datasets for Insightful Visualizations and Analysis

This report explores mastering web scraping Zomato datasets to generate insightful visualizations and perform in-depth analysis for data-driven decisions.

Case Studies

View More

Case Study - Revolutionizing Global Tire Business with Tyre Pricing and Market Intelligence

Leverage tyre pricing and market intelligence to gain a competitive edge, optimize strategies, and drive growth in the global tire industry.

Case Study: Data Scraping for Ferry and Cruise Price Optimization

Explore how data scraping optimizes ferry schedules and cruise prices, providing actionable insights for businesses to enhance offerings and pricing strategies.

Infographics

View More

How to Use Web Scraping for Extracting Costco Product Specifications?

Web scraping enables businesses to access and analyze detailed product specifications from Costco, including prices, descriptions, availability, and reviews. By leveraging this data, companies can gain insights into customer preferences, monitor competitor pricing, and optimize their product offerings for better market performance.

Web Scraping Best Buy Data – A Complete Tutorial

Learn how to effectively scrape data from Best Buy, including product details, pricing, reviews, and stock information, using tools like Selenium and Beautiful Soup.