TripAdvisor Restaurant Web Data Extraction with Python
With Web Scraping Services , we can scrape Tripadvisor restaurants data information like product ratings, prices, and other data from various websites. We can later utilize this data for many applications like research, data analytics, data science, and business intelligence. In Python, businesses often try Tripadvisor restaurant data scraping using libraries like Scrapy, BeautifulSoup, and Request, which simplify parsing and retrieving web data.
But hold on, this isn't just another data extraction project.
This data scraping project aims to extract restaurant information from any location worldwide by adding experimental data analytics of the retrieved data.
What does a CSV file of scraped TripAdvisor Restaurant data look like?
The extracted restaurant data will include
- Restaurant Name
- Star/Bubble Rating
- Total Customer Reviews
- Cuisines
You can also find other information like data offset, restaurant serial number, and page number.
Input Parameters- Control Variables
In this project, we've opted to scrape data from Berlin, Germany-based restaurants. You can choose any city using the TripAdvisor filter option and get the link according to your preferences. If you want to scrape Bangalore-based restaurants, you'll get the link like this https://www.tripadvisor.in/Restaurants-g297628-Bengaluru_Bangalore_District_Karnataka.html with 297628 as a geolocation code. If we dive deeper, we observe over 11 thousand restaurants in Bengaluru. Therefore our input variables will be
- City Name
- Geolocation Code
- Upper Data Offset
Since we've selected the location with the geolocation code, let's begin to execute the script. The primary step is to import and install needed Python libraries. Then it comes to defining control variables. Considering that we're scraping Berlin-based restaurants, we'll define control variables accordingly. Further, there are around thirty restaurants listed on each page on the source platform, which matches our page size. On the last page, there are over 6300 data offsets near our upper limit of data offsets.
We'll see changes in control variables as per our target city.
We'll use around ten functions in our script, as below.
- get_url
- get_card
- get_soup_content
- parse_tripadvisor
- get_restaurant_data_from_card
- scrape_title
- scrape_cuisines>
- scrape_reviews
- save_to_csv
Let's briefly explore each function.
Get_url
It takes data offset, geocode, and city name in the input field and makes a different link for each page you want to scrape. The link follows a data offset pattern in the multiple of 30 as below.
- https://www.tripadvisor.in/Restaurants-g187323-oa90-Berlin.html#EATERY_LIST_CONTENTS
- https://www.tripadvisor.in/Restaurants-g187323-oa120-Berlin.html#EATERY_LIST_CONTENTS
- https://www.tripadvisor.in/Restaurants-g187323-oa150-Berlin.html#EATERY_LIST_CONTENTS
get_soup_content
It takes data offset, geocode, and city name as input and calls get_url. The function also creates a response object using the generated link. After accessing HTML, we should parse it and load it to BS4 format. This soup function handily enables us to use valuable information like ratings, cuisines, ratings, etc.
get_card
This function helps get restaurant cards as per the serial number or count of the restaurant. You can see the card tags in the below example image.
Parse_tripadvisor
This function takes already defined steps from earlier steps as the input. It is one of the essential functions in our script. Variables city_name, page_size, data_offset_upper_limit, data_offset_lower_limit, page_num and geo_code take values from the scraping_control_variables directory. You'll see that data_offset_current and data_offset_lower_limit have the same values with increments of 30 on each page. The while loop keeps running till it scrapes the last page. Page_start_offset and page_end_offset take values in sets of 30 in each step. As every page usually includes thirty restaurants. But considering we can't completely assure whether every page contains less than 30 restaurants, we have added the if condition in the loop. The function get_restaurant_data_from_card scrapes restaurant details and adds them to the empty list.
get_restaurant_data_from_card
It takes the page number, current data offset, restaurant count, and card number in input and calls each scrape function generated to collect restaurant information.
Data Extraction Functions to Collect Restaurant Data
Every function below takes the card as input which includes all data related to a specific restaurant.
- scrape_star_ratings (gets customer ratings of restaurants.)
- scrape_reviews (gets the review count of restaurants)
- scrape_cuisines (gets all restaurant cuisines)
- scrape_title (gets restaurant names)
Title Tag
Rating Tag
Reviews Tag
Cuisines Tag
Storing the extracted file in CSV format
Finally, let us store the data in CSV format in our local database. You can use this CSB format for data science and data analytics projects.
Script Output
We are yet to finish the process!
Let's try some experimental data analysis in the extracted data, where we'll plot the following study with Seaborn.
- The Top ten most famous cuisines in Berlin
- Number of star ratings bs review of Berlin-based restaurants
The clean_dataframe function cleans the data frame of scraped output, like splitting serial numbers from names of restaurants, splitting cuisines, dropping useless columns, and removing unwanted noise from a few columns.
The scatter_plot_viz function makes a bar graph of famous cuisines in Berlin with the help of Seaborne. It displays the best restaurants in Berlin by studying the relationship between review counts and ratings. Per the graph, we'd prefer the restaurant with more reviews and quality ratings.
The popular_cuisinws function generates bar graphs for the most famous cuisines by collecting a dataset of cuisine counts. To take cuisine count, we'll need to split every cousin separately and split them into individual rows.
We have shared How to Scrape TripAdvisor Restaurant data for Any City with Python and also explained how we can use this restaurant data scraping ahead in data analytics and market research. If you still have any doubts, contact our team at Actowiz Solutions.
