Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
Actowiz Metrics Now Live!
logo
Unlock Smarter , Faster Analytics!
GeoIp2\Model\City Object
(
    [raw:protected] => Array
        (
            [city] => Array
                (
                    [geoname_id] => 4509177
                    [names] => Array
                        (
                            [de] => Columbus
                            [en] => Columbus
                            [es] => Columbus
                            [fr] => Columbus
                            [ja] => コロンバス
                            [pt-BR] => Columbus
                            [ru] => Колумбус
                            [zh-CN] => 哥伦布
                        )

                )

            [continent] => Array
                (
                    [code] => NA
                    [geoname_id] => 6255149
                    [names] => Array
                        (
                            [de] => Nordamerika
                            [en] => North America
                            [es] => Norteamérica
                            [fr] => Amérique du Nord
                            [ja] => 北アメリカ
                            [pt-BR] => América do Norte
                            [ru] => Северная Америка
                            [zh-CN] => 北美洲
                        )

                )

            [country] => Array
                (
                    [geoname_id] => 6252001
                    [iso_code] => US
                    [names] => Array
                        (
                            [de] => USA
                            [en] => United States
                            [es] => Estados Unidos
                            [fr] => États Unis
                            [ja] => アメリカ
                            [pt-BR] => EUA
                            [ru] => США
                            [zh-CN] => 美国
                        )

                )

            [location] => Array
                (
                    [accuracy_radius] => 20
                    [latitude] => 39.9625
                    [longitude] => -83.0061
                    [metro_code] => 535
                    [time_zone] => America/New_York
                )

            [postal] => Array
                (
                    [code] => 43215
                )

            [registered_country] => Array
                (
                    [geoname_id] => 6252001
                    [iso_code] => US
                    [names] => Array
                        (
                            [de] => USA
                            [en] => United States
                            [es] => Estados Unidos
                            [fr] => États Unis
                            [ja] => アメリカ
                            [pt-BR] => EUA
                            [ru] => США
                            [zh-CN] => 美国
                        )

                )

            [subdivisions] => Array
                (
                    [0] => Array
                        (
                            [geoname_id] => 5165418
                            [iso_code] => OH
                            [names] => Array
                                (
                                    [de] => Ohio
                                    [en] => Ohio
                                    [es] => Ohio
                                    [fr] => Ohio
                                    [ja] => オハイオ州
                                    [pt-BR] => Ohio
                                    [ru] => Огайо
                                    [zh-CN] => 俄亥俄州
                                )

                        )

                )

            [traits] => Array
                (
                    [ip_address] => 216.73.216.101
                    [prefix_len] => 22
                )

        )

    [continent:protected] => GeoIp2\Record\Continent Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [code] => NA
                    [geoname_id] => 6255149
                    [names] => Array
                        (
                            [de] => Nordamerika
                            [en] => North America
                            [es] => Norteamérica
                            [fr] => Amérique du Nord
                            [ja] => 北アメリカ
                            [pt-BR] => América do Norte
                            [ru] => Северная Америка
                            [zh-CN] => 北美洲
                        )

                )

            [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                (
                    [0] => en
                )

            [validAttributes:protected] => Array
                (
                    [0] => code
                    [1] => geonameId
                    [2] => names
                )

        )

    [country:protected] => GeoIp2\Record\Country Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [geoname_id] => 6252001
                    [iso_code] => US
                    [names] => Array
                        (
                            [de] => USA
                            [en] => United States
                            [es] => Estados Unidos
                            [fr] => États Unis
                            [ja] => アメリカ
                            [pt-BR] => EUA
                            [ru] => США
                            [zh-CN] => 美国
                        )

                )

            [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                (
                    [0] => en
                )

            [validAttributes:protected] => Array
                (
                    [0] => confidence
                    [1] => geonameId
                    [2] => isInEuropeanUnion
                    [3] => isoCode
                    [4] => names
                )

        )

    [locales:protected] => Array
        (
            [0] => en
        )

    [maxmind:protected] => GeoIp2\Record\MaxMind Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                )

            [validAttributes:protected] => Array
                (
                    [0] => queriesRemaining
                )

        )

    [registeredCountry:protected] => GeoIp2\Record\Country Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [geoname_id] => 6252001
                    [iso_code] => US
                    [names] => Array
                        (
                            [de] => USA
                            [en] => United States
                            [es] => Estados Unidos
                            [fr] => États Unis
                            [ja] => アメリカ
                            [pt-BR] => EUA
                            [ru] => США
                            [zh-CN] => 美国
                        )

                )

            [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                (
                    [0] => en
                )

            [validAttributes:protected] => Array
                (
                    [0] => confidence
                    [1] => geonameId
                    [2] => isInEuropeanUnion
                    [3] => isoCode
                    [4] => names
                )

        )

    [representedCountry:protected] => GeoIp2\Record\RepresentedCountry Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                )

            [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                (
                    [0] => en
                )

            [validAttributes:protected] => Array
                (
                    [0] => confidence
                    [1] => geonameId
                    [2] => isInEuropeanUnion
                    [3] => isoCode
                    [4] => names
                    [5] => type
                )

        )

    [traits:protected] => GeoIp2\Record\Traits Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [ip_address] => 216.73.216.101
                    [prefix_len] => 22
                    [network] => 216.73.216.0/22
                )

            [validAttributes:protected] => Array
                (
                    [0] => autonomousSystemNumber
                    [1] => autonomousSystemOrganization
                    [2] => connectionType
                    [3] => domain
                    [4] => ipAddress
                    [5] => isAnonymous
                    [6] => isAnonymousProxy
                    [7] => isAnonymousVpn
                    [8] => isHostingProvider
                    [9] => isLegitimateProxy
                    [10] => isp
                    [11] => isPublicProxy
                    [12] => isResidentialProxy
                    [13] => isSatelliteProvider
                    [14] => isTorExitNode
                    [15] => mobileCountryCode
                    [16] => mobileNetworkCode
                    [17] => network
                    [18] => organization
                    [19] => staticIpScore
                    [20] => userCount
                    [21] => userType
                )

        )

    [city:protected] => GeoIp2\Record\City Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [geoname_id] => 4509177
                    [names] => Array
                        (
                            [de] => Columbus
                            [en] => Columbus
                            [es] => Columbus
                            [fr] => Columbus
                            [ja] => コロンバス
                            [pt-BR] => Columbus
                            [ru] => Колумбус
                            [zh-CN] => 哥伦布
                        )

                )

            [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                (
                    [0] => en
                )

            [validAttributes:protected] => Array
                (
                    [0] => confidence
                    [1] => geonameId
                    [2] => names
                )

        )

    [location:protected] => GeoIp2\Record\Location Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [accuracy_radius] => 20
                    [latitude] => 39.9625
                    [longitude] => -83.0061
                    [metro_code] => 535
                    [time_zone] => America/New_York
                )

            [validAttributes:protected] => Array
                (
                    [0] => averageIncome
                    [1] => accuracyRadius
                    [2] => latitude
                    [3] => longitude
                    [4] => metroCode
                    [5] => populationDensity
                    [6] => postalCode
                    [7] => postalConfidence
                    [8] => timeZone
                )

        )

    [postal:protected] => GeoIp2\Record\Postal Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [code] => 43215
                )

            [validAttributes:protected] => Array
                (
                    [0] => code
                    [1] => confidence
                )

        )

    [subdivisions:protected] => Array
        (
            [0] => GeoIp2\Record\Subdivision Object
                (
                    [record:GeoIp2\Record\AbstractRecord:private] => Array
                        (
                            [geoname_id] => 5165418
                            [iso_code] => OH
                            [names] => Array
                                (
                                    [de] => Ohio
                                    [en] => Ohio
                                    [es] => Ohio
                                    [fr] => Ohio
                                    [ja] => オハイオ州
                                    [pt-BR] => Ohio
                                    [ru] => Огайо
                                    [zh-CN] => 俄亥俄州
                                )

                        )

                    [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                        (
                            [0] => en
                        )

                    [validAttributes:protected] => Array
                        (
                            [0] => confidence
                            [1] => geonameId
                            [2] => isoCode
                            [3] => names
                        )

                )

        )

)
 country : United States
 city : Columbus
US
Array
(
    [as_domain] => amazon.com
    [as_name] => Amazon.com, Inc.
    [asn] => AS16509
    [continent] => North America
    [continent_code] => NA
    [country] => United States
    [country_code] => US
)
How-to-Scrape-10-Online-Shops-within-30-Minutes-Using-Scrapy-and

Get sourced data you want to kick-start an App project

Get-sourceddata-you-want-to-kick-start-an-App-project
  • You are a developer
  • You would love to create a wonderful Web Application
  • You are completely dedicated towards your project

Although you tick all these points, you still require a domain associated dataset before you write a one line code. It is because contemporary applications use a huge amount of data at the same time or in batches for providing value for users.

In this blog, we will explain our workflow to generate such datasets. You would see how we deal with automated data scraping of different websites with no manual intervention.

Our objective is to produce a dataset to make a Price Comparison WebApp. A product category we will be utilizing as an example is hand- bags. For this application, price and product data of hand-bags need to be collected from various online-sellers every day. Though some sellers offer an API to access all the required details, not all follow the similar route. Therefore, using web scrapping is certain!

In this example, we will create web spiders for 10 sellers using Scrapy and Python. Then, we will automate the procedure using Apache Airflow and there will be no requirement for manual involvements to execute the whole procedure periodically.

A Live Demo Web App with Source Code

You can get all associated source code in the GitHub repository.

Our Web Scraping Workflow

Before we start any web data scraping project, we need to define which sites will get covered in this project. We have decided to include 10 websites that are the most stayed online stores in Turkey for hand bags. You can observe them in our GitHub repository.

Step1: Install Scrapy and Set Up Project Folders

Step1-Install-Scrapy-and-Set-Up-Project-Folders

You need to install Scrapy in your computer and create a Scrapy project before making any Scrapy spiders.

Project Files & Folders

Project-Files-&-Folders

We made a folder structure in the local computer for neatly placing project files in separate folders.

A ‘csvFiles’ folder has a CSV file for all websites extracted. Spiders would be reading from the CSV files to find ‘starting URLs’ for initiating scraping as we do not need to hard-code them in spiders.

‘fashionWebScraping’ folder has Scrapy spiders with helper scripts including ‘pipelines.py’, ‘settings.py’, and ‘item.py’. We need to modify a few of Scrapy helper scripts for executing the scraping procedure successfully.

Product images extracted will get saved in an ‘images_scraped’ folder.

Product-images-extracted-will

During the procedure of web data scraping, all the product data like pricing, name, product links and image links would be saved in JSON files within ‘jsonFiles’ folder.

There would be utility scripts to execute some tasks like;

  • ‘deldub.py’ for detecting and removing duplicate product data in JSON files after data extraction ends.
  • ‘deleteFiles.py’ for deleting all the JSON files produced at prior scrapping session.
  • ‘jsonPrep.py’ is one more utility script for detecting and deleting null line objects in JSON files after data extraction ends.
  • ‘jsonToes.py’ for populating ElasticSearch clusters in the remote location reading from JSON files. It provides a full-text real-time search experience.
  • ‘sitemap_gen.py’ is to generate a site map that covers different product links.

Step2: Understand Particular Site’s URL Structure with Settling CSV Files for Preliminary URLs

Step2-Understand-Particular-Sites-URL-Structure-with-Settling-CSV-Files-for-Preliminary-URLs

After creation of project folders, the next step is populating the CSV files with starting URLs for every website we like to extract.

Nearly every e-commerce site provides pagination for navigating users through product list. Each time you navigate for next page, a page parameter within URL increases. Just go through the example URL given below, where a ‘page’ parameter gets used.

We will utilize {} placeholder to iterate URLs by incrementing values of ‘page’. We will utilize a ‘gender’ column within CSV file for defining gender categories of a particular URL.

Therefore, the last CSV file would look like that:

The similar principles applied to rest of sites in a project.

Therefore-the-last-CSV

Step3: Modify ‘settings.py’ and ‘items.py’

Step3-Modify-settings-py-and

Step 1: Installing and Setting Up packages

To do web scraping, we need to modify ‘items.py’ for defining ‘item objects’ that are used for storing the extracted data.

To describe general output data formats Scrapy offers an Item class. These item objects are easy containers used for collecting the extracted data. They offer dictionary-like APIs with an easy syntax to declare their accessible fields.

using scrapy.org

using-scrapy-org

After that, we need to modify ‘settings.py’. It is necessary to customize image pipelines and spiders’ behavior.

These Scrapy settings permit you in customizing the behavior of different Scrapy components like the core, pipelines, extensions, and spiders.

using scrapy.org

‘settings.py’ and ‘item.py’ are valid for different spiders in the project.

Step4: Making Spiders

Step4-Making-Spiders

Spiders from Scrapy are the classes that define how certain sites (or groups of websites) will get scraped, together with how to do crawling (i.e. follow links) as well as how to scrape structured data using their pages (i.e. extracting items). Spiders are a place where you can define crawling’s custom behavior and parsing the pages for any particular website (or in a few cases, one group of websites).

using scrapy.org

The given shell command makes a clear spider file. It’s time to write codes in the fashionBOYNER.py file:

The-given-shell-command-makes-a-clear

The spider class has 2 functions including ‘start_requests’ as well as ‘parse_product_pages’.

In function ‘start_requests’, we read from definite CSV file that we have already produced to get starting URL data. Then we repeat the placeholder {} for passing URLs of product pages into a ‘parse_product_pages’ function.

We could also pass ‘gender’ meta-data into ‘parse_product_pages’ function with ‘Request’ method using ‘meta={‘gender’: row[‘gender’]}’ stricture.

we-could-see-status-of-the-we-could-see-status-of-the

In ‘parse_product_pages’ function, we do the real web extraction and populate Scrapy items using the extracted data.

We use Xpath for locating HTML sections containing product data on a web page.

The initial Xpath expression given scrapes the entire product listing from current pages getting scrapped. All the necessary product data is contained within ‘div’ content elements.

The-initial-Xpath-expression

We have to loop in ‘content’ for reaching individual products as well as storing them in Scrapy items. Using XPath expressions, we could easily find the essential HTML elements in ‘content’.

We-have-to-loop-in-content-for-reaching

We have to loop in

Step 5: Run Spiders and Store the Extracted Data in the JSON File

Step-5-Run-Spiders-and-Store-the-Extracted-Data-in-the-JSON-File

With this scraping procedure, every product item is saved in the JSON file. Every website has a particular JSON file occupied with data in every spider run.

Use of jsonlines format can be more memory-efficient in comparison to JSON format, particularly if you scrape many web pages at one session.

Note that a JSON file name begins with ‘rawdata’ indicating that next step is checking and validating the extracted raw data before utilizing them in the application.

Step 6: Clean and Validate the Extracted Data in the JSON Files

Step-6-Clean-and-Validate-the-Extracted-Data-in-the-JSON-Files

After the extraction procedure ends, you might have some items you need to remove from JSON files, before utilizing them in the application.

You might have some line items having duplicate values or null fields. Both cases need a correction procedure which we handle using ‘deldub.py’ and ‘jsonPrep.py’.

‘jsonPrep.py’ is looking for line items having null values as well as removes them if detected. You could find a code having explanations given below:

The results are saved with the file name begins with ‘prepdata’ in ‘jsonFiles’ folder after null line items get removed.

The-results-are-saved-with-the-file-name

‘deldub.py’ needs duplicate line items as well as removes them if detected. You could find a code having explanations given below:

‘deldub.py’ needs duplicate line items as well as removes them if detected. You could find a code having explanations given below:

Automate the Entire Scraping Workflow Using an Apache Airflow

When we define the scraping procedure, we can jump into workflow automation. We will utilize Apache Airflow that is a Python-based automation tool made by Airbnb.

We will offer terminal commands to install and configure Apache Airflow.

Generating a DAG file

In the Airflow, a DAG (Directed Acyclic Graph) is the collection of different tasks you need to run, well-organized in the way which reflects their dependencies and relationships.

For instance, an easy DAG can include three jobs: A, B, & C. This might indicate that A need to successfully run before B could run, however, C could run anytime. This indicates that job A times out afterwards 5 minutes, and B could get restarted around 5 times if it fails. This might also indicate that workflow would run each night at 10 pm, however shouldn’t begin until any certain date.

DAG’s that are defined in the Python file, is to organize a task flow. We would not define real tasks within a DAG file.

Let’s make a DAG folder with an empty Python file and start defining workflow using Python codes.

Let-s-make-a-DAG-folder-with-an

Many operators are there given by Airflow for describing the job within a DAG file. We have listed commonly utilized ones given below.

implements-a-bash-command sends-emails sends-HTTP-requests waits-for-certain-files-time-database-rows-S3-key-and-more

We are planning to utilize only ‘BashOperator’ as we would be completing different tasks using Python scripts.

We-are-planning-to-utilize-only

By following the tutorial, we generated bash scripts to do every task. You could find them in the Github repository.

To begin a DAG workflow, we have to run an Airflow Scheduler. It will execute a scheduler using a specified configuration in the ‘airflow.cfg’ file. A scheduler monitors every task in every DAG positioned in a ‘dags’ folder as well as triggers the task execution if dependencies are met.

When we run an airflow scheduler, we could see status of the tasks through visiting http://0.0.0.0:8080 on the browser. Airflow offers a user interface in which we could see and observe scheduled dags.

When-we-run-an-airflow

Conclusion

We have shown here the web scraping workflow from starting till end.

Hopefully, it will assist you grasp the fundamentals of web scrapping with workflow automation.

For more details, contact Actowiz Solutions. You can also reach us for all your mobile app scraping and web scraping services requirements.

GeoIp2\Model\City Object
(
    [raw:protected] => Array
        (
            [city] => Array
                (
                    [geoname_id] => 4509177
                    [names] => Array
                        (
                            [de] => Columbus
                            [en] => Columbus
                            [es] => Columbus
                            [fr] => Columbus
                            [ja] => コロンバス
                            [pt-BR] => Columbus
                            [ru] => Колумбус
                            [zh-CN] => 哥伦布
                        )

                )

            [continent] => Array
                (
                    [code] => NA
                    [geoname_id] => 6255149
                    [names] => Array
                        (
                            [de] => Nordamerika
                            [en] => North America
                            [es] => Norteamérica
                            [fr] => Amérique du Nord
                            [ja] => 北アメリカ
                            [pt-BR] => América do Norte
                            [ru] => Северная Америка
                            [zh-CN] => 北美洲
                        )

                )

            [country] => Array
                (
                    [geoname_id] => 6252001
                    [iso_code] => US
                    [names] => Array
                        (
                            [de] => USA
                            [en] => United States
                            [es] => Estados Unidos
                            [fr] => États Unis
                            [ja] => アメリカ
                            [pt-BR] => EUA
                            [ru] => США
                            [zh-CN] => 美国
                        )

                )

            [location] => Array
                (
                    [accuracy_radius] => 20
                    [latitude] => 39.9625
                    [longitude] => -83.0061
                    [metro_code] => 535
                    [time_zone] => America/New_York
                )

            [postal] => Array
                (
                    [code] => 43215
                )

            [registered_country] => Array
                (
                    [geoname_id] => 6252001
                    [iso_code] => US
                    [names] => Array
                        (
                            [de] => USA
                            [en] => United States
                            [es] => Estados Unidos
                            [fr] => États Unis
                            [ja] => アメリカ
                            [pt-BR] => EUA
                            [ru] => США
                            [zh-CN] => 美国
                        )

                )

            [subdivisions] => Array
                (
                    [0] => Array
                        (
                            [geoname_id] => 5165418
                            [iso_code] => OH
                            [names] => Array
                                (
                                    [de] => Ohio
                                    [en] => Ohio
                                    [es] => Ohio
                                    [fr] => Ohio
                                    [ja] => オハイオ州
                                    [pt-BR] => Ohio
                                    [ru] => Огайо
                                    [zh-CN] => 俄亥俄州
                                )

                        )

                )

            [traits] => Array
                (
                    [ip_address] => 216.73.216.101
                    [prefix_len] => 22
                )

        )

    [continent:protected] => GeoIp2\Record\Continent Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [code] => NA
                    [geoname_id] => 6255149
                    [names] => Array
                        (
                            [de] => Nordamerika
                            [en] => North America
                            [es] => Norteamérica
                            [fr] => Amérique du Nord
                            [ja] => 北アメリカ
                            [pt-BR] => América do Norte
                            [ru] => Северная Америка
                            [zh-CN] => 北美洲
                        )

                )

            [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                (
                    [0] => en
                )

            [validAttributes:protected] => Array
                (
                    [0] => code
                    [1] => geonameId
                    [2] => names
                )

        )

    [country:protected] => GeoIp2\Record\Country Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [geoname_id] => 6252001
                    [iso_code] => US
                    [names] => Array
                        (
                            [de] => USA
                            [en] => United States
                            [es] => Estados Unidos
                            [fr] => États Unis
                            [ja] => アメリカ
                            [pt-BR] => EUA
                            [ru] => США
                            [zh-CN] => 美国
                        )

                )

            [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                (
                    [0] => en
                )

            [validAttributes:protected] => Array
                (
                    [0] => confidence
                    [1] => geonameId
                    [2] => isInEuropeanUnion
                    [3] => isoCode
                    [4] => names
                )

        )

    [locales:protected] => Array
        (
            [0] => en
        )

    [maxmind:protected] => GeoIp2\Record\MaxMind Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                )

            [validAttributes:protected] => Array
                (
                    [0] => queriesRemaining
                )

        )

    [registeredCountry:protected] => GeoIp2\Record\Country Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [geoname_id] => 6252001
                    [iso_code] => US
                    [names] => Array
                        (
                            [de] => USA
                            [en] => United States
                            [es] => Estados Unidos
                            [fr] => États Unis
                            [ja] => アメリカ
                            [pt-BR] => EUA
                            [ru] => США
                            [zh-CN] => 美国
                        )

                )

            [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                (
                    [0] => en
                )

            [validAttributes:protected] => Array
                (
                    [0] => confidence
                    [1] => geonameId
                    [2] => isInEuropeanUnion
                    [3] => isoCode
                    [4] => names
                )

        )

    [representedCountry:protected] => GeoIp2\Record\RepresentedCountry Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                )

            [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                (
                    [0] => en
                )

            [validAttributes:protected] => Array
                (
                    [0] => confidence
                    [1] => geonameId
                    [2] => isInEuropeanUnion
                    [3] => isoCode
                    [4] => names
                    [5] => type
                )

        )

    [traits:protected] => GeoIp2\Record\Traits Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [ip_address] => 216.73.216.101
                    [prefix_len] => 22
                    [network] => 216.73.216.0/22
                )

            [validAttributes:protected] => Array
                (
                    [0] => autonomousSystemNumber
                    [1] => autonomousSystemOrganization
                    [2] => connectionType
                    [3] => domain
                    [4] => ipAddress
                    [5] => isAnonymous
                    [6] => isAnonymousProxy
                    [7] => isAnonymousVpn
                    [8] => isHostingProvider
                    [9] => isLegitimateProxy
                    [10] => isp
                    [11] => isPublicProxy
                    [12] => isResidentialProxy
                    [13] => isSatelliteProvider
                    [14] => isTorExitNode
                    [15] => mobileCountryCode
                    [16] => mobileNetworkCode
                    [17] => network
                    [18] => organization
                    [19] => staticIpScore
                    [20] => userCount
                    [21] => userType
                )

        )

    [city:protected] => GeoIp2\Record\City Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [geoname_id] => 4509177
                    [names] => Array
                        (
                            [de] => Columbus
                            [en] => Columbus
                            [es] => Columbus
                            [fr] => Columbus
                            [ja] => コロンバス
                            [pt-BR] => Columbus
                            [ru] => Колумбус
                            [zh-CN] => 哥伦布
                        )

                )

            [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                (
                    [0] => en
                )

            [validAttributes:protected] => Array
                (
                    [0] => confidence
                    [1] => geonameId
                    [2] => names
                )

        )

    [location:protected] => GeoIp2\Record\Location Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [accuracy_radius] => 20
                    [latitude] => 39.9625
                    [longitude] => -83.0061
                    [metro_code] => 535
                    [time_zone] => America/New_York
                )

            [validAttributes:protected] => Array
                (
                    [0] => averageIncome
                    [1] => accuracyRadius
                    [2] => latitude
                    [3] => longitude
                    [4] => metroCode
                    [5] => populationDensity
                    [6] => postalCode
                    [7] => postalConfidence
                    [8] => timeZone
                )

        )

    [postal:protected] => GeoIp2\Record\Postal Object
        (
            [record:GeoIp2\Record\AbstractRecord:private] => Array
                (
                    [code] => 43215
                )

            [validAttributes:protected] => Array
                (
                    [0] => code
                    [1] => confidence
                )

        )

    [subdivisions:protected] => Array
        (
            [0] => GeoIp2\Record\Subdivision Object
                (
                    [record:GeoIp2\Record\AbstractRecord:private] => Array
                        (
                            [geoname_id] => 5165418
                            [iso_code] => OH
                            [names] => Array
                                (
                                    [de] => Ohio
                                    [en] => Ohio
                                    [es] => Ohio
                                    [fr] => Ohio
                                    [ja] => オハイオ州
                                    [pt-BR] => Ohio
                                    [ru] => Огайо
                                    [zh-CN] => 俄亥俄州
                                )

                        )

                    [locales:GeoIp2\Record\AbstractPlaceRecord:private] => Array
                        (
                            [0] => en
                        )

                    [validAttributes:protected] => Array
                        (
                            [0] => confidence
                            [1] => geonameId
                            [2] => isoCode
                            [3] => names
                        )

                )

        )

)
 country : United States
 city : Columbus
US
Array
(
    [as_domain] => amazon.com
    [as_name] => Amazon.com, Inc.
    [asn] => AS16509
    [continent] => North America
    [continent_code] => NA
    [country] => United States
    [country_code] => US
)

Start Your Project

+1

Additional Trust Elements

✨ "1000+ Projects Delivered Globally"

⭐ "Rated 4.9/5 on Google & G2"

🔒 "Your data is secure with us. NDA available."

💬 "Average Response Time: Under 12 hours"

From Raw Data to Real-Time Decisions

All in One Pipeline

Scrape Structure Analyze Visualize

Look Back Analyze historical data to discover patterns, anomalies, and shifts in customer behavior.

Find Insights Use AI to connect data points and uncover market changes. Meanwhile.

Move Forward Predict demand, price shifts, and future opportunities across geographies.

Industry:

Coffee / Beverage / D2C

Result

2x Faster

Smarter product targeting

★★★★★

“Actowiz Solutions has been instrumental in optimizing our data scraping processes. Their services have provided us with valuable insights into our customer preferences, helping us stay ahead of the competition.”

Operations Manager, Beanly Coffee

✓ Competitive insights from multiple platforms

Industry:

Real Estate

Result

2x Faster

Real-time RERA insights for 20+ states

★★★★★

“Actowiz Solutions provided exceptional RERA Website Data Scraping Solution Service across PAN India, ensuring we received accurate and up-to-date real estate data for our analysis.”

Data Analyst, Aditya Birla Group

✓ Boosted data acquisition speed by 3×

Industry:

Organic Grocery / FMCG

Result

Improved

competitive benchmarking

★★★★★

“With Actowiz Solutions' data scraping, we’ve gained a clear edge in tracking product availability and pricing across various platforms. Their service has been a key to improving our market intelligence.”

Product Manager, 24Mantra Organic

✓ Real-time SKU-level tracking

Industry:

Quick Commerce

Result

2x Faster

Inventory Decisions

★★★★★

“Actowiz Solutions has greatly helped us monitor product availability from top three Quick Commerce brands. Their real-time data and accurate insights have streamlined our inventory management and decision-making process. Highly recommended!”

Aarav Shah, Senior Data Analyst, Mensa Brands

✓ 28% product availability accuracy

✓ Reduced OOS by 34% in 3 weeks

Industry:

Quick Commerce

Result

3x Faster

improvement in operational efficiency

★★★★★

“Actowiz Solutions' data scraping services have helped streamline our processes and improve our operational efficiency. Their expertise has provided us with actionable data to enhance our market positioning.”

Business Development Lead,Organic Tattva

✓ Weekly competitor pricing feeds

Industry:

Beverage / D2C

Result

Faster

Trend Detection

★★★★★

“The data scraping services offered by Actowiz Solutions have been crucial in refining our strategies. They have significantly improved our ability to analyze and respond to market trends quickly.”

Marketing Director, Sleepyowl Coffee

Boosted marketing responsiveness

Industry:

Quick Commerce

Result

Enhanced

stock tracking across SKUs

★★★★★

“Actowiz Solutions provided accurate Product Availability and Ranking Data Collection from 3 Quick Commerce Applications, improving our product visibility and stock management.”

Growth Analyst, TheBakersDozen.in

✓ Improved rank visibility of top products

Trusted by Industry Leaders Worldwide

Real results from real businesses using Actowiz Solutions

★★★★★
'Great value for the money. The expertise you get vs. what you pay makes this a no brainer"
Thomas Gallao
Thomas Galido
Co-Founder / Head of Product at Upright Data Inc.
Product Image
2 min
★★★★★
“I strongly recommend Actowiz Solutions for their outstanding web scraping services. Their team delivered impeccable results with a nice price, ensuring data on time.”
Thomas Gallao
Iulen Ibanez
CEO / Datacy.es
Product Image
1 min
★★★★★
“Actowiz Solutions offered exceptional support with transparency and guidance throughout. Anna and Saga made the process easy for a non-technical user like me. Great service, fair pricing highly recommended!”
Thomas Gallao
Febbin Chacko
-Fin, Small Business Owner
Product Image
1 min

See Actowiz in Action – Real-Time Scraping Dashboard + Success Insights

Blinkit (Delhi NCR)

In Stock
₹524

Amazon USA

Price Drop + 12 min
in 6 hrs across Lel.6

Appzon AirPdos Pro

Price
Drop −12 thr

Zepto (Mumbai)

Improved inventory
visibility & planning

Monitor Prices, Availability & Trends -Live Across Regions

Actowiz's real-time scraping dashboard helps you monitor stock levels, delivery times, and price drops across Blinkit, Amazon: Zepto & more.

✔ Scraped Data: Price Insights Top-selling SKUs

Our Data Drives Impact - Real Client Stories

Blinkit | India (Retail Partner)

"Actowiz's helped us reduce out of stock incidents by 23% within 6 weeks"

✔ Scraped Data, SKU availability, delivery time

US Electronics Seller (Amazon - Walmart)

With hourly price monitoring, we aligned promotions with competitors, drove 17%

✔ Scraped Data, SKU availability, delivery time

Zepto Q Commerce Brand

"Actowiz's helped us reduce out of stock incidents by 23% within 6 weeks"

✔ Scraped Data, SKU availability, delivery time

Actowiz Insights Hub

Actionable Blogs, Real Case Studies, and Visual Data Stories -All in One Place

All
Blog
Case Studies
Infographics
Report
Aug 25, 2025

Starbucks Menu Price Fluctuation - Price Analysis of Starbucks Items in New York and LA

Track Starbucks Menu Price Fluctuation in New York and LA. Analyze latte, frappuccino, and cappuccino prices from 2020–2025 for smarter pricing and promotions.

thumb

Getaround Vehicle Availability API Improved Chicago Car Access by 30%

Discover how Getaround Vehicle Availability API helped a Chicago mobility app optimize real-time car access and increase bookings by 30% effectively.

thumb

Price Optimization vs Price Monitoring - 20% Margin Boost Revealed in 2025 Market Insights

Price Optimization vs Price Monitoring reveals a 20% margin boost in 2025, offering insights to maximize profitability and pricing strategy.

Aug 25, 2025

Starbucks Menu Price Fluctuation - Price Analysis of Starbucks Items in New York and LA

Track Starbucks Menu Price Fluctuation in New York and LA. Analyze latte, frappuccino, and cappuccino prices from 2020–2025 for smarter pricing and promotions.

Aug 24, 2025

Trending Discounts on Personal Care Products in Australia - Weekly Woolworths vs Coles Price Comparison

Compare weekly discounts on personal care products in Australia with our Woolworths vs Coles Price Comparison—stay updated, save money, and shop smart every week!

Aug 23, 2025

Benefits of Monthly Angi and Zillow Scraping for Service Aggregators

Unlock actionable insights with monthly Angi and Zillow scraping, helping service aggregators track trends, analyze competitors, and optimize business strategies.

thumb

Getaround Vehicle Availability API Improved Chicago Car Access by 30%

Discover how Getaround Vehicle Availability API helped a Chicago mobility app optimize real-time car access and increase bookings by 30% effectively.

thumb

D2C Fashion Inventory Optimization Using Demand Data from Naver

Discover how a D2C brand improved inventory management and sales with Naver demand data, leveraging D2C Fashion Inventory Optimization strategies effectively.

thumb

Cosmetic Pricing Intelligence in Daejeon Using Naver Product Scraping

Explore how Actowiz Solutions enabled Cosmetic Pricing Intelligence in Daejeon with Naver product scraping, delivering insights for smarter retail strategies.

thumb

Price Optimization vs Price Monitoring - 20% Margin Boost Revealed in 2025 Market Insights

Price Optimization vs Price Monitoring reveals a 20% margin boost in 2025, offering insights to maximize profitability and pricing strategy.

thumb

Skin Type-Based Skincare Insights in Korea Using Naver Beauty Categories

Explore skin type-based skincare insights in Korea using Naver Beauty Categories, uncovering trends, product preferences, and consumer behavior across categories.

thumb

AI-Powered Web Scraping Market Analysis 2025–2035 – Forecasts, Competitors & Use Cases

Explore the AI-Powered Web Scraping Market Analysis 2025–2035, including growth forecasts, key competitors, use cases, and emerging industry trends.