How to Extract Amazon Product Data using BeautifulSoup?

Web Extraction Services

To create a service using data scraping, you may need to go through IP Blocking and proxy management. So, it’s good to understand underlying processes and technologies; however, for bulk data scraping, you can deal with scraping API service providers like Actowiz Solutions. They will take care of JavaScript and Ajax requests for all dynamic pages.

Basic Requirements:

To make a soup, you need suitable ingredients. Similarly, our latest web scraper needs certain gears.

Python: The easily usable and massive assembly of libraries makes Python “Matchless” for extracting websites. Although if a user does not get it pre-installed, then refer here.

Beautiful Soup: It is one of the data scraping libraries of Python. The clean and easy usage of a library makes that a top candidate for data scraping. After successfully installing Python, a user could install BeautifulSoup using:

pip install bs4

Web Browser: As we need to toss out many unnecessary details from a site, we require particular tags and ids to filter them out. So, a web browser like Mozilla Firefox or Google Chrome serves the objective of finding those tags.

Making a User-Agent

Many websites follow specific protocols to block robots from retrieving data. So, to scrape data from the script, we want to make a User-Agent, a string that tells a server about the kind of host sending a request.

The website has tons of user agents to select. Here is an example of the User-Agent having a header value.

HEADERS = ({'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36', 'Accept-Language': 'en-US, en;q=0.5'})

There is one extra field with HEADERS named “Accept-Language” that translates a webpage into English-US if required.

Send a Request to the URL

A URL (Uniform Resource Locator) accesses its webpage. Using the URL, we send a request to a webpage to access its data.

URL = "https://www.amazon.com/Sony-PlayStation-Pro-1TB-Console-4/dp/B07K14XKZH/" webpage = requests.get(URL, headers=HEADERS)

The demanded webpage features one Amazon product. Therefore, our Python script scrapes product information like “The Current Price”, “The Product Name Product,” etc.

Create a Soup of Data

The webpage variable has a response established by a website. We pass the content of a reply and parser types to a Beautiful Soup function.

soup = BeautifulSoup(webpage.content, "lxml")

lxml is the high-speed parser used by BeautifulSoup for breaking down an HTML page into complex Python objects. Usually, there are four types of Python Objects available:

Tag -It resembles XML or HTML tags that include names with attributes.

NavigableString -It reaches the stored text within the tag.

BeautifulSoup -The whole parsed document

Comments -Lastly, the leftover pieces of an HTML page are not comprised of the three categories.

Discover Particular Tags for Object Extraction

Among the most exciting parts of the project is unearthing the tags and ids storing relevant data. As mentioned earlier, we are using web browsers to accomplish the task.

We open a webpage in a browser and review the relevant elements by pressing the right-click.

Consequently, a panel opens on the screen's right-hand side, as shown in the figure.

When we get tag values, scraping data becomes very easy. However, we must study certain functions well-defined for BeautifulSoup objects.

Extract Product Title

Using the find() function accessible to particular search tags with precise attributes, we find the Tag Object having a product title.

# Outer Tag Object title = soup.find("span", attrs={"id":'productTitle'})

After that, we take the NavigableString Object

# Inner NavigableString Object title_value = title.string

And lastly, we strip additional spaces and convert an object to the string value.

# Title as a string value title_string = title_value.strip()

Then, we can observe at types of variables with type() function.

# Printing types of values for efficient understanding print(type(title)) print(type(title_value)) print(type(title_string)) print() # Printing Product Title print("Product Title = ", title_string)

Output:

Using the find() function accessible to particular search tags with precise attributes, we find the Tag Object having a product title.

Product Title = Sony PlayStation 4 Pro 1TB Console - Black (PS4 Pro)

Similarly, we have to find tag values for product information like “Consumer Ratings” and “Price of a Product.”

Python Script for Scraping Product Data

The following Python script displays the following details for a product:

Output:

Using the find() function accessible to particular search tags with precise attributes, we find the Tag Object having a product title.

Product Title = Sony PlayStation 4 Pro 1TB Console - Black (PS4 Pro) Product Price = $473.99 Product Rating = 4.7 out of 5 stars Number of Product Reviews = 1,311 ratings Availability = In Stock.

Now, as we understand how to scrape information from one Amazon webpage, we could apply a similar script to different web pages by changing a URL.

Furthermore, let us try and fetch links from the Amazon search result webpage.

Python Script for Scraping Product Data across Different Webpages

Here is the complete Python script to list different PlayStation deals.

Output:

The given Python script is not limited to a PlayStations list. We can change a URL to any other link to the Amazon search result, like earphones or headphones.

As stated before, the tags and layout of the HTML page might change over time, making the above code useless. The reader must bring home a concept of data scraping and methods learned in this blog.

Conclusion

Web Scraping has many benefits, including “product price comparison” to “consumer tendency analysis.” As the internet is available to everybody and Python is a straightforward language, anybody can do Web Scraping to meet their requirements.

We hope this blog is easy to understand for you. For more details, contact Actowiz Solutions now! Please comment with your thoughts for any feedback or queries.