How to Get the Finest Deals Using Web Scraping on a Raspberry Pi Device?

Fundamentals

For the purpose of our use cases, a laptop or a Raspberry Pi can be used, but we will be using Raspberry Pi as a web scraping server that runs continuously. There are numerous Raspberry Pi projects available online, but most of them require some electrical engineering.

Python 3 is the language of choice for our application. It has a wide range of powerful libraries, and it is easy to get started and create a prototype. Since Python 2 will no longer be supported from January 1st, 2020, we will use Python 3.

Scrapy is among the finest open-source web extraction frameworks available in Python. It is a powerful and incredibly fast tool that is at the core of our set of tools. While new versions have been developed, the core components have remained largely unchanged. We will be using the latest version of Scrapy 2.0.1 on Python 3.6.10 in this article.

To inspect objects and extract HTML tags with ease, a modern browser with developer tools enabled is recommended.

Website

To succeed in web scraping, it's important to choose a site with a high amount of traffic. Some websites that offer discounts and promo codes include SlickDeals, Dealnews, and DealMoon. For the purposes of this blog, we will be using SlickDeals as our chosen website to scrape data. While there will be different components on the HTML to extract, there are no restrictions on choosing a website that aligns with your interests.

Use a Developer Tool for Scraping Context

1. Go to SlickDeals website

2. To find the best bargains, check out the Frontpage Slickdeals section. Here, each item is accompanied by a product image, title, store/website, original price, current price, likes, and shipping details.

3. To extract data using Python's loop, start by opening the developer tool on the browser or inspecting an element on the website. Most developer tools will highlight your selection and focus on the HTML tag you choose. Look for a similar pattern to use in your loop. If you move to the next item, you may see the same tag again. For instance, a div tag with class "fpItem" is used for each item in this example - < div class="fpItem" >.

4. To retrieve additional data related to < div class="fpItem" >, we need to access its parent. You can obtain the names of all classes by following the same steps described earlier with the use of Developer Tools in your browser and extracting the necessary fields.

Scrapy Code

Once you have determined the appropriate class from which to extract data, you can create a Python Scrapy project and execute a test run. For additional information on Scrapy, please visit the following link.

The code shown is a file named spider.py located in Scrapy's Spider folder. To begin, we name the crawler "slickdeals." As previously mentioned, we use Selector to obtain a list item by calling it.

After obtaining the list, we can go through each item and gather the necessary information by utilizing XPath. We will verify if the class includes our desired keyword during this process.

After collecting the data, we save it in a CSV file for further analysis. If you prefer, you may also send an email with a specific keyword using Python's email module. Here's an example code without any content.

To test this program using a project root directory, just execute

And the result will look something like this and you’d observe the fields which we have extracted.

Schedule

To ensure our program runs continuously, it's best to use an energy-efficient Raspberry Pi. Once the code is confirmed to work, we can schedule the web crawler application to run automatically using Linux's crontab feature. To do this, open crontab with the command "crontab -e" and add the following command: "*/15 * * * *". This will execute the web crawler every 15 minutes.

Final thought

Great job! Your web scraping program is now up and running 24/7, just as you requested. Whether your aim is to find great deals, freebies, or coupons, our program is working tirelessly in the background to monitor and alert you of the best finds. We hope this blog has given you some insight into web scraping and the potential to build even more advanced programs on a small device like the Raspberry Pi.

For more details, contact Actowiz Solutions! You can also tell us your about your mobile app scraping or web scraping service requirements.