How to Scrape Hotel Prices Data Using Web Scraping?

Requirements

We require Python 3.x here and we assume that you have installed it on your PC. Together with that, you have to install a couple of more libraries that will be further used in this blog for web data scraping.

Requests would help us in making an HTTP connection using Bing.

BeautifulSoup will assist us to make an HTML tree for smoother data extraction.

Setup

Initially, make a folder and install libraries given above.

Within this folder, make a Python file in which will write a code. Here are the given data points which we will extract from the targeted website.

• Name
• Address
• Prices
• Ratings
• Type of Room
• Facilities

Let’s Extract Booking.com

As all the things are set, let’s make GET requests to the targeted website to observe if that works.

The code is very easy and requires no description but let us explain you a bit. Initially, we have imported two libraries which we have downloaded earlier in the blog and then we have declared headers and targeted URLs. Lastly, we made the GET request to a targeted URL. When you print, you can see 200 codes otherwise your codes are not right.

How to Extract Data Points?

As we have decided which data points we will extract, lets get the HTML location through inspecting chrome.

For this blog, we would be using find() and find_all() techniques of BeautifulSoup to get targeted elements. DOM structure would decide which technique will be better for every element.

Scraping Hotel Name & Address

Let’s review chrome and get the DOM location of name and address.

As you observe hotel name is available under the h2 tag having class pp- header__title. For simplicity let’s initially make a soup variable using a BeautifulSoup constructor and from it we will scrape all data points.

Here, the BS4 will utilize an HTML parser for converting complex HTML documents into complex tree of Python objects. So, let’s use a soup variable to scrape name and address.

In the similar manner, we would scrape the address.

The property address is stored under a span tag having a class name called hp_address_subtitle.

Scrape Facilities and Ratings

Again we will review and find a DOM location of the facilities and rating element.

Ratings are stored under a div tag having class d10a6220b4. We would use the similar soup variable for scraping this element. The given code will scrape the rating data.

Scraping facilities is somewhat tricky. We will make a list where we will store the facility HTML elements. Then, we will work a for loop to repeat over the elements and save individual text in main array.

Let’s observe how that can be done with two easy steps:

fac variable would hold all facility elements. Now, it’s time to scrape them one by one.

fac_arr array would store all text values for elements. We have effectively managed to scrape the key facilities.

Scrape Room Types and Prices

This section is the most complicated part of this tutorial. The DOM structure for booking.com is somehow complex and requires detailed study before scraping room type and pricing information.

The tbody tag here has all the data. With tbody, you would get tr tag as the tag holds all data from the initial column.

After going a step down, you would find differnt td tags where data like price, Room Type, etc. are easily available.

Initially, let’s get all tr tags.

One point noticeable here is that all tr tags have data-block-id attributes. Let’s gather all these ids into a list.

Now, as you get all ids rest of a job becomes somewhat easier. We will repeat over all data-block-ids to scrape room types and room pricing from separate tr blocks.

allData variable would store all HTML data for any specific data-block-id.

Now, we could move to td tags which could be available inside the tr tag. Let’s scrape rooms data first.

Here is the fun part, if you have multiple options for a specific room type then you need to use similar room for next pricing set in a loop. Let us explain it to you using a picture.

Here, we get three prices for one type of room. Therefore, when for a loop iterates the value of rooms, the variable would be None. You could see that by printing it. Therefore, we will utilize the old values of rooms till we get the new values. We hope you are getting our point.

Here, the last_room would store last value of the rooms till we get the new value.

It’s time to scrape the pricing now.

Pricing gets stored under a div tag having class

Let’s utilize allData variables to get it and scrape the text.

Finally, we have managed to extract all data elements which we are interested in.

Entire Code

You can scrape other information like reviews, amenities, etc. You only need to make some changes and you would be able to scrape them also. Together with this, you can scrape other hotel data by changing unique names of a hotel in its URL.

So, the code would look like.

The output of this script should look like this.

Why Scrape Booking.com Data? – The Benefits

Many travel agencies gather a huge amount of data from competitor’s sites. They understand if they need to get an edge over the competitors, they must get access to the competitor’s price strategies.

To get advantages over a niche competitor has to extract different websites and aggregate data. Then to finish adjust own prices after doing comparison with them. Get discounts or show how cheap the prices are together with the competitor’s prices.

As there are over 200 OTAs available in the market. it becomes difficult to extract and compare. We would advise to utilize service like hotel search API for getting the prices of hotels in any city across the globe.

Conclusion

Clearly, hotel data extraction goes beyond that and it was only an example about how Python could be used to scrape Booking.com for price comparison objectives. You can utilize Python to scrape other sites like Hotels.com, Expedia, etc.

However, scraping at scale might not be possible at this procedure. After some time the booking.com would block your IPs and data pipeline would get blocked forever. For continuous scraping usage of Web Scraping API that will rotate the IPs on all new requests and would use headless chrome for reducing any chances of blockage.

For more information, contact Actowiz Solutions now! You can also reach us for your mobile app scraping and web scraping services requirements.