Government Schemes Data Scraping: Central & State Program Intelligence

About the Client

Location: Mumbai, India

Industry: Public Data Analytics & Policy Research

Objective: To collect, validate, and structure current Central and State Government schemes across key sectors — Healthcare, Education, Agriculture, and Business Development — from verified government sources and portals.

The client required an up-to-date, accurate, and comprehensive dataset for both Central and State-specific welfare programs. The focus was on ongoing schemes only, excluding outdated or discontinued ones.

The Challenge

Public sector data is often scattered across multiple portals, PDFs, and departmental websites. For India, this fragmentation occurs between:

• Central Government portals like mygov.in, india.gov.in, pib.gov.in, and gov.in/ministry pages.
• State Government websites, which vary widely in format, structure, and data freshness.

Key Challenges Faced:

• No Centralized Source: Each ministry and state maintains its own scheme directory. Cross-referencing them manually is time-consuming.
• Dynamic Updates: Schemes get renamed, merged, or discontinued — scraping must adapt to frequent updates.
• PDF and Table-Based Data: Many schemes are listed in unstructured formats (PDF, HTML tables, or scanned images).
• Data Validation: Ensuring the scheme is active, not closed, and verifiable from official sites required careful filtering logic.
• Categorization: Assigning each scheme to relevant sectors (Healthcare, Education, Agriculture, Business, etc.) for analytics-ready output.

Project Goals

Actowiz Solutions' mandate was clear:

• Extract live, verifiable data of current Central and State government schemes.
• Classify them by sector, target audience, and funding type.
• Provide dataset exports in CSV, JSON, and Excel formats.
• Build a framework for periodic updates (weekly or monthly).
• Generate high-level visual insights (counts by sector, state, and level).

Scraping Architecture

[ Government Websites (Central & State) ]

↓

[ Scrapy Spider + Playwright Automation ]

↓

[ HTML & PDF Parsing (Titles, Descriptions, URLs) ]

↓

[ NLP-based Keyword Categorization (Healthcare / Education / etc.) ]

↓

[ Validation & Deduplication ]

↓

[ Structured Export (CSV, JSON, MySQL) ]

↓

[ Power BI Dashboard Visualization ]

Infographic

Actowiz Data Quality Process

Actowiz Solutions built custom data validation modules to ensure reliability:

• Active Status Check: Cross-verifies scheme names with recent mentions on official portals (last 6 months).
• Duplicate Merging: Merges identical schemes listed under multiple ministries (e.g., PM-KISAN under Agriculture and Finance).
• Text Cleaning: Removes stopwords, symbols, and redundant details for clean database storage.
• Categorization via NLP: Machine-learning-assisted keyword tagging for accurate sector classification.
• URL Verification: Confirms every scheme's source is from a .gov.in or .nic.in domain.

Key Insights Delivered

• Sectoral Trends: Agriculture and healthcare programs had the widest geographical spread across states.
• Funding Patterns: Over 60% of active schemes were centrally funded but state-implemented.
• Target Demographics: The highest number of schemes targeted farmers (31%), followed by students (22%), and MSMEs (15%).
• Regional Concentration: Maharashtra, Tamil Nadu, Telangana, and Gujarat led in state-run initiatives.

Impact for the Client

• Unified Access to Scattered Data – One dataset covering verified government schemes across ministries and states.
• Saved 150+ Hours of Manual Research – Automated scraper now updates data weekly.
• Decision Support for Research Analysts – Enabled fast comparison of central and state-level initiatives.
• Dashboard-Ready Dataset – Integrated with Power BI for instant visualization of live updates.

Client Testimonial

“Actowiz Solutions turned a difficult, fragmented research task into a structured data system. Their scraping and validation accuracy were exceptional, and we now have an updated dashboard tracking all active schemes weekly.”

— Head of Policy Analytics, Mumbai-based Consultancy

Compliance & Ethics

Scraping limited to publicly accessible .gov.in and .nic.in domains.

No personal or confidential data collected.

Compliant with Indian IT Act and data usage policies.

Data used strictly for public research and analytics.

Actowiz Solutions follows ethical web scraping practices and ensures data accuracy and compliance at every stage.

Why Actowiz Solutions

• Expertise in public sector data intelligence and government data scraping.
• Proven tools for dynamic content, PDF parsing, and NLP-driven classification.
• End-to-end solutions: scraping → cleaning → categorization → dashboard visualization.
• Ongoing support and update automation for long-term data accuracy.

Future Enhancements

• Add new schemes dynamically using scheduled Playwright crawls every 72 hours.
• Integrate social impact metrics (beneficiaries count, fund allocation).
• Regional language support for Hindi, Tamil, Telugu, and Marathi scheme descriptions.
• API access for real-time scheme lookup.
• AI summarization module to auto-generate scheme briefs.

Conclusion

This case study demonstrates how Actowiz Solutions transformed the complex, decentralized landscape of Indian government schemes into an organized, real-time dataset for decision-makers.

• Reliable visibility into ongoing Central and State programs.
• Faster access to welfare and business initiatives.
• Actionable data for policy research and business planning.

Through cutting-edge scraping technology, data validation, and classification, Actowiz Solutions helped the client gain:

With automation and NLP-driven classification, the client now maintains a live, weekly-updated dashboard of verified schemes — ensuring informed decisions and transparent analytics.