About the Client

Location: Mumbai, India

Industry: Public Data Analytics & Policy Research

Objective: To collect, validate, and structure current Central and State Government schemes across key sectors — Healthcare, Education, Agriculture, and Business Development — from verified government sources and portals.

The client required an up-to-date, accurate, and comprehensive dataset for both Central and State-specific welfare programs. The focus was on ongoing schemes only, excluding outdated or discontinued ones.

The Challenge

Public sector data is often scattered across multiple portals, PDFs, and departmental websites. For India, this fragmentation occurs between:

Central Government portals like mygov.in, india.gov.in, pib.gov.in, and gov.in/ministry pages.
State Government websites, which vary widely in format, structure, and data freshness.

Key Challenges Faced:

No Centralized Source: Each ministry and state maintains its own scheme directory. Cross-referencing them manually is time-consuming.
Dynamic Updates: Schemes get renamed, merged, or discontinued — scraping must adapt to frequent updates.
PDF and Table-Based Data: Many schemes are listed in unstructured formats (PDF, HTML tables, or scanned images).
Data Validation: Ensuring the scheme is active, not closed, and verifiable from official sites required careful filtering logic.
Categorization: Assigning each scheme to relevant sectors (Healthcare, Education, Agriculture, Business, etc.) for analytics-ready output.

Project Goals

Actowiz Solutions' mandate was clear:

Extract live, verifiable data of current Central and State government schemes.
Classify them by sector, target audience, and funding type.
Provide dataset exports in CSV, JSON, and Excel formats.
Build a framework for periodic updates (weekly or monthly).
Generate high-level visual insights (counts by sector, state, and level).

Data Sources Identified

Source	Description	Type
india.gov.in	Official government directory of Central schemes	Central
mygov.in	Citizen engagement and scheme announcements	Central
pib.gov.in	Official Press Information Bureau releases	Central
State Government Portals	Schemes per state (maharashtra.gov.in, tamilnadugov.in, up.gov.in, etc.)	State
Ministry Sites	Agriculture, MSME, Education, Health, Finance, Women & Child Development	Sectoral
News & Gazette Updates	Scheme launches and updates	Cross-source validation

Technology Stack

Component	Tools Used
Web Scraping	Python (Scrapy + BeautifulSoup + Requests-HTML)
Dynamic Rendering	Playwright (for JavaScript-heavy sites)
Data Parsing	Regex, Pandas
Data Storage	MySQL + CSV + JSON
Validation Engine	Rule-based filters for "active" schemes
Dashboard Visualization	Power BI / Tableau
Automation	Cron jobs for weekly updates

Scraping Architecture

[ Government Websites (Central & State) ]

↓

[ Scrapy Spider + Playwright Automation ]

↓

[ HTML & PDF Parsing (Titles, Descriptions, URLs) ]

↓

[ NLP-based Keyword Categorization (Healthcare / Education / etc.) ]

↓

[ Validation & Deduplication ]

↓

[ Structured Export (CSV, JSON, MySQL) ]

↓

[ Power BI Dashboard Visualization ]

Data Extraction Fields

Field	Description
Scheme Name	Official scheme title
Type	Central / State
State / Ministry	Applicable entity
Category	Healthcare, Education, Agriculture, Business
Description	Summary of benefits
Target Group	Farmers, Students, Entrepreneurs, Women, MSMEs, etc.
Launch Year	Year of introduction
Current Status	Active / Merged / Suspended
Official Link	Source URL for validation

Sample Dataset (Extracted Snapshot)

Scheme Name	Type	Sector	Target Group	Description	Source
Ayushman Bharat Pradhan Mantri Jan Arogya Yojana	Central	Healthcare	Low-income families	Health insurance coverage up to ₹5 lakh per family per year.	https://pmjay.gov.in
PM-KISAN Samman Nidhi	Central	Agriculture	Small & marginal farmers	Direct income support of ₹6,000 annually in three installments.	https://pmkisan.gov.in
Startup India Seed Fund Scheme	Central	Business	Startups / Entrepreneurs	Early-stage funding support for startups across sectors.	https://startupindia.gov.in
Samagra Shiksha Abhiyan	Central	Education	School students	Integrated education scheme for holistic school development.	https://education.gov.in
Mahatma Jyotirao Phule Jan Arogya Yojana	State (Maharashtra)	Healthcare	Residents of Maharashtra	Free healthcare for families below income threshold.	https://jeevandayee.gov.in
Rythu Bandhu Scheme	State (Telangana)	Agriculture	Farmers	Investment support for each crop season at ₹10,000/acre.	https://rythubandhu.telangana.gov.in

Infographic

Chart: Number of Schemes by Sector (Sample Visualization)

Sector	Total Schemes (Approx.)
Healthcare	42
Education	38
Agriculture	55
Business & Industry	33
Women & Child Development	20
Skill & Employment	27

Insight: Agriculture and Healthcare remain the most active sectors with the highest number of ongoing initiatives in 2024–2025.

Actowiz Data Quality Process

Actowiz Solutions built custom data validation modules to ensure reliability:

Active Status Check: Cross-verifies scheme names with recent mentions on official portals (last 6 months).
Duplicate Merging: Merges identical schemes listed under multiple ministries (e.g., PM-KISAN under Agriculture and Finance).
Text Cleaning: Removes stopwords, symbols, and redundant details for clean database storage.
Categorization via NLP: Machine-learning-assisted keyword tagging for accurate sector classification.
URL Verification: Confirms every scheme's source is from a .gov.in or .nic.in domain.

Results

Metric	Achieved
Total Schemes Extracted	215+ (across 24 states and 1 UT)
Verified Active Schemes	180+
Central Schemes	95
State Schemes	85
Average Update Cycle	Weekly (Automated)
Data Accuracy	98.6% validated

Implementation Timeline

Phase	Duration	Activities
Discovery & Source Mapping	2 Days	Identified verified central & state sources
Scraper Development	5 Days	Built Scrapy + Playwright hybrid crawler
Data Extraction & Cleaning	3 Days	Parsed, validated, and normalized data
QA & Output Formatting	2 Days	Validated schema and removed duplicates
Dashboard Setup	2 Days	Visualization in Power BI
Total Duration	~12 Days	End-to-end delivery

Key Insights Delivered

Sectoral Trends: Agriculture and healthcare programs had the widest geographical spread across states.
Funding Patterns: Over 60% of active schemes were centrally funded but state-implemented.
Target Demographics: The highest number of schemes targeted farmers (31%), followed by students (22%), and MSMEs (15%).
Regional Concentration: Maharashtra, Tamil Nadu, Telangana, and Gujarat led in state-run initiatives.

Impact for the Client

Unified Access to Scattered Data – One dataset covering verified government schemes across ministries and states.
Saved 150+ Hours of Manual Research – Automated scraper now updates data weekly.
Decision Support for Research Analysts – Enabled fast comparison of central and state-level initiatives.
Dashboard-Ready Dataset – Integrated with Power BI for instant visualization of live updates.

Client Testimonial

“Actowiz Solutions turned a difficult, fragmented research task into a structured data system. Their scraping and validation accuracy were exceptional, and we now have an updated dashboard tracking all active schemes weekly.”

— Head of Policy Analytics, Mumbai-based Consultancy

Compliance & Ethics

Scraping limited to publicly accessible .gov.in and .nic.in domains.

No personal or confidential data collected.

Compliant with Indian IT Act and data usage policies.

Data used strictly for public research and analytics.

Actowiz Solutions follows ethical web scraping practices and ensures data accuracy and compliance at every stage.

Why Actowiz Solutions

Expertise in public sector data intelligence and government data scraping.
Proven tools for dynamic content, PDF parsing, and NLP-driven classification.
End-to-end solutions: scraping → cleaning → categorization → dashboard visualization.
Ongoing support and update automation for long-term data accuracy.

Future Enhancements

Add new schemes dynamically using scheduled Playwright crawls every 72 hours.
Integrate social impact metrics (beneficiaries count, fund allocation).
Regional language support for Hindi, Tamil, Telugu, and Marathi scheme descriptions.
API access for real-time scheme lookup.
AI summarization module to auto-generate scheme briefs.

Conclusion

This case study demonstrates how Actowiz Solutions transformed the complex, decentralized landscape of Indian government schemes into an organized, real-time dataset for decision-makers.

Reliable visibility into ongoing Central and State programs.
Faster access to welfare and business initiatives.
Actionable data for policy research and business planning.

Through cutting-edge scraping technology, data validation, and classification, Actowiz Solutions helped the client gain:

With automation and NLP-driven classification, the client now maintains a live, weekly-updated dashboard of verified schemes — ensuring informed decisions and transparent analytics.

Hear It Directly from Our Clients

Watch how businesses like yours are using Actowiz data to drive growth.

▶

1 min

★★★★★

"Actowiz Solutions offered exceptional support with transparency and guidance throughout. Anna and Saga made the process easy for a non-technical user like me. Great service, fair pricing!"

Thomas Galido

Co-Founder / Head of Product at Upright Data Inc.

▶

2 min

★★★★★

"Actowiz delivered impeccable results for our company. Their team ensured data accuracy and on-time delivery. The competitive intelligence completely transformed our pricing strategy."

Iulen Ibanez

CEO / Datacy.es

▶

1:30

★★★★★

"What impressed me most was the speed — we went from requirement to production data in under 48 hours. The API integration was seamless and the support team is always responsive."

Febbin Chacko

-Fin, Small Business Owner