How to Scrape TripAdvisor Hotels Data Using Python and BeautifulSoup?

Import Different Libraries

# Import the libraries.
import requests
from bs4 import BeautifulSoup
import pandas as pd
import csv

Review a Webpage’s HTML Structure

We have to recognize the contents and structure of HTML tags in webpages. For that project, we would be using TripAdvisor Hawaii Hotels & Places of staying webpage (given below). You can get this webpage through choosing a link.

We can extract this webpage through parsing HTML of a page and scraping the data required for the dataset. To extract some data from the web page, just right-click anywhere on this webpage, choose inspect from a drop-down list and click an arrow icon given on the screen’s upper left-hand side with HTML and click on hotel name (Prince Waikiki) in review section of a webpage. It will result in the given screen shown.

On HTML screen, you would see highlighted an HTML line having the Hotel Name called Prince Waikiki.

If you are moving one line from the tag then you would find a div tag having the class of “listing_title”. It is a parent of tag. Therefore, if you want to find, scrape, and capture hotel names on a webpage you might follow these steps.

Get all HTML lines for any particular parent (div tag having class = listing_title) that might include their related children.

Scrape data elements and create a list having all the hotel names.

The code to find and extract hotel names might be the following:

hotels = []
for name in soup.findAll('div',{'class':'listing_title'}):
hotels.append(name.text.strip())

We will get, scrape and store other data elements on a webpage following similar procedures as given above.

Find and Scrape Data Elements

For all data elements we need to scrape, we will get all HTML lines, which are within any particular class and tag. Then, we will scrape data elements as well as store data in the list.

# Find and extract data elements.
hotels = []
for name in soup.findAll('div',{'class':'listing_title'}):
    hotels.append(name.text.strip())
ratings = []
for rating in soup.findAll('a',{'class':'ui_bubble_rating'}):
    ratings.append(rating['alt'])
reviews = []
for review in soup.findAll('a',{'class':'review_count'}):
    reviews.append(review.text.strip())
prices = []
for p in soup.findAll('div',{'class':'price-wrap'}):
    prices.append(p.text.replace('₹','').strip())

Creating a Data Frame

We would create a dictionary, which will have data names and standards for all data elements which were scraped.

# Create the dictionary.
dict = {'Hotel Names':hotels,'Ratings':ratings,'Number of Reviews':reviews,'Prices':prices}

Create and show a data frame.

# Create the dataframe.
hawaii = pd.DataFrame.from_dict(dict)
hawaii.head(10)

Converting Data Frames into a CSV file

# Convert dataframe to CSV file.
hawaii.to_csv('hotels.csv', index=False, header=True)

Making it all together…

# Import the libraries.
import requests
from bs4 import BeautifulSoup
import pandas as pd
import csv

# Extract the HTML and create a BeautifulSoup object.
url = ('https://www.tripadvisor.in/Hotels-g28932-Hawaii-Hotels.html')

user_agent = ({'User-Agent':
			'Mozilla/5.0 (Windows NT 10.0; Win64; x64) \
			AppleWebKit/537.36 (KHTML, like Gecko) \
			Chrome/90.0.4430.212 Safari/537.36',
			'Accept-Language': 'en-US, en;q=0.5'})

def get_page_contents(url):
    page = requests.get(url, headers = user_agent)
    return BeautifulSoup(page.text, 'html.parser')

soup = get_page_contents(url)

# Find and extract the data elements.
hotels = []
for name in soup.findAll('div',{'class':'listing_title'}):
    hotels.append(name.text.strip())

ratings = []
for rating in soup.findAll('a',{'class':'ui_bubble_rating'}):
    ratings.append(rating['alt'])  

reviews = []
for review in soup.findAll('a',{'class':'review_count'}):
    reviews.append(review.text.strip())

prices = []
for p in soup.findAll('div',{'class':'price-wrap'}):
    prices.append(p.text.replace('₹','').strip())  

# Create the dictionary.
dict = {'Hotel Names':hotels,'Ratings':ratings,'Number of Reviews':reviews,'Prices':prices}

# Create the dataframe.
hawaii = pd.DataFrame.from_dict(dict)
hawaii.head(10)

# Convert dataframe to CSV file.
hawaii.to_csv('hotels.csv', index=False, header=True)

Thank you so much to read this blog. Please give your valuable comments or feedback. For the best mobile app scraping and web scraping services, contact Actowiz Solutions now!