Scraping JavaScript-Intensive Websites Like an Expert Using Python

JavaScript has become ubiquitous on the web, posing a challenge for web scraping. While most data is easily accessible in the HTML of a page, there are instances where the data is only available after the JavaScript is executed and rendered. This complicates the scraping process.

In previous articles, we have covered web scraping using user-friendly Python libraries. However, these methods need to be improved when dealing with JavaScript rendering. We need to employ more advanced tools and techniques to tackle websites that hide their data behind JavaScript rendering.

Best Tool for this Job

Normally, we would suggest some go-to libraries to do web scraping:

BeautifulSoup

Requests

While the previously mentioned tools can handle various tasks, they cannot render JavaScript. Fortunately, there is a dedicated category of tools designed specifically for this purpose called Browser Automation tools.

Browser Automation tools are built to simulate and automate the web browsing experience, allowing tasks to be executed at intervals or speeds surpassing human capabilities. While these tools are commonly used for website testing by owners, they also offer the functionality required to render JavaScript and scrape the underlying data.

These tools provide:

A comprehensive solution.

Combining JavaScript rendering capabilities with web scraping functionality.

Making them ideal for extracting data from JavaScript-intensive websites.

By leveraging Browser Automation tools, you can effectively overcome the challenge of scraping websites that rely heavily on JavaScript.

Among the popular tools are:

Scrapy
Playwright
Selenium

For the purpose of this example, let's delve into using Selenium. Additionally, we will utilize the reliable BeautifulSoup library to parse the response and extract the desired data.

Set up a Workspace

To fully automate a web browser, some additional setup is necessary beyond the basic installation of libraries. We will need to install the following components:

Chrome Browser (or any other web browser of your choice, but we will use Chrome for this example).

ChromeDriver: This is the web driver specific to Chrome that allows interaction with the browser programmatically.

To follow along, you can install Chrome on your system. As for ChromeDriver installation, we can use a convenient Python library that handles the installation for us.

With that in mind, let's proceed with installing the required libraries.

Once they get installed, we could start importing:

To simplify the installation process, we can use the chromedriver_autoinstaller library, which automatically installs ChromeDriver and adds it to the system's PATH if it's not already present. This saves us some effort and can be achieved with a single line of code:

By executing this code, the library will handle the installation of ChromeDriver seamlessly.

Here's a summary of the steps we've taken to set up our environment. Assuming you have already installed Python, you can use pip to install the necessary libraries. After installing selenium, bs4, and chromedriver-autoinstaller, your Python file should look something like this:

Get a Webpage

Now that we have our environment set up, we can start making web page requests. To accomplish this, we need to configure the WebDriver object that Selenium will use. Here's an example of how you can set it up:

Now we can instruct the webdriver to retrieve a web page. For this example, we'll scrape Rotten Tomatoes Certified Fresh Movies.

Although the data we want (movie titles, ratings, etc.) can be obtained without rendering JavaScript, it's much easier to parse when it's rendered.

This page heavily relies on JavaScript, as shown in the JavaScript-enabled site:

And using JavaScript disabled:

We can ask for this web page using driver object “get” method

And we could get an html output with a page_source attribute:

Just to do recap, let’s go through the code:

Parse the HTML

Now, Selenium can handle parsing the data, but in most cases, we'll rely on BeautifulSoup for parsing the HTML. Let's create a BeautifulSoup object from the page source:

Now, we need to have something which will look like this:

If we were to print the soup object we've created, we would see the entire web page, excluding some of the fancy formatting. Fortunately, in this case, we don't have to wait for JavaScript execution.

However, in some scenarios, we may need to wait for JavaScript execution. This can be achieved either through Implicit Waiting or Explicit Waiting.

Since we don't need to worry about that here, let's focus on finding the information we're interested in:

It appears that all the movies we're interested in are contained within div elements with the class "mb-movie". Each of these divs contains information about an individual movie.

To extract the relevant information, we can use BeautifulSoup's find_all() method with the appropriate parameters:

We can have each of them and find a title, release date, and score and easily using BeautifulSoup:

Conclusion

In summary, we have accomplished several tasks in a short period of time:

Installed Chrome and ChromeDriver.

Used a Python library to install ChromeDriver automatically.

Fetched a web page that heavily relies on JavaScript using Selenium.

Parsed and extracted data from the web page using BeautifulSoup.

Here's a final overview of our progress, with the data printed in the terminal:

For more information, contact Actowiz Solutions now! Call us for all your mobile app scaping and data collection service requirements.

Hear It Directly from Our Clients

Watch how businesses like yours are using Actowiz data to drive growth.

▶

1 min

★★★★★

"Actowiz Solutions offered exceptional support with transparency and guidance throughout. Anna and Saga made the process easy for a non-technical user like me. Great service, fair pricing!"

Febbin Chacko

Small Business Owner

Fin

▶

2 min

★★★★★

"Actowiz delivered impeccable results for our company. Their team ensured data accuracy and on-time delivery. The competitive intelligence completely transformed our pricing strategy."

Javier Ibanez

Head of Analytics

atacy.es

▶

1:30

★★★★★

"What impressed me most was the speed — we went from requirement to production data in under 48 hours. The API integration was seamless and the support team is always responsive."

Rajesh Kumar

CTO

QComm Brand

⭐ 4.8/5 Average Rating

📹 50+ Video Testimonials

🔄 92% Client Retention

🌍 50+ Countries Served

AI Solutions Engineered
for Your Needs

→

LLM-Powered Attribute Extraction: High-precision product matching using large language models for accurate data classification.

→

Advanced Computer Vision: Fine-grained object detection for precise product classification using text and image embeddings.

→

GPT-Based Analytics Layer: Natural language query-based reporting and visualization for business intelligence.

→

Human-in-the-Loop AI: Continuous feedback loop to improve AI model accuracy over time.

🎯 Product Matching 🏷️ Attribute Tagging 📝 Content Optimization 💬 Sentiment Analysis 📊 Prompt-Based Reporting

Connect the Dots Across
Your Retail Ecosystem

We partner with agencies, system integrators, and technology platforms to deliver end-to-end solutions across the retail and digital shelf ecosystem.

Analytics Services

Ad Tech

Price Optimization

Business Consulting

System Integration

Market Research

Become a Partner →

Array ( [as_domain] => amazon.com [as_name] => Amazon.com, Inc. [asn] => AS16509 [continent] => North America [continent_code] => NA [country] => United States [country_code] => US )

Enterprise Data Extraction

AI Training Data

Analytics Dashboard

AI Dynamic Pricing

Social Commerce

Ready-to-Use Data

AI Training Data

Ready-Made Scrapers

API Playground

Case Studies

Sample Datasets

ROI Calculator

Scraping JavaScript-Intensive Websites Like an Expert Using Python

May 16, 2023

Best Tool for this Job

Set up a Workspace

Get a Webpage

Parse the HTML

Conclusion

Trusted by Global Leaders Across Q-Commerce, Travel, Retail, and FoodTech

Hear It Directly from Our Clients

Join 3,000+ Companies Growing with Actowiz

Why Global Leaders Trust Actowiz

AI Solutions Engineeredfor Your Needs

Connect the Dots AcrossYour Retail Ecosystem

Popular Datasets — Ready to Download

Latest Insights & Resources

How IHG Hotels & Resorts Data Scraping Helps Overcome Real-Time Availability and Rate Monitoring Issues

UK Grocery Chain Achieves 300% ROI on Promotional Campaigns

Track UK Grocery Products Daily Using Automated Data Scraping to Monitor 50,000+ UK Grocery Products from Morrisons, Asda, Tesco, Sainsbury’s, Iceland, Co-op, Waitrose, Ocado

Start Where It Makes Sense for You

Our perks are irreplaceable

Time Zone Flexibility

Clear Communication

Uncompromising Quality

Our Affiliations

Request Free Sample Data

AI Solutions Engineered
for Your Needs

Connect the Dots Across
Your Retail Ecosystem