AI-Powered Walk-In Drive Job Scraping & Recruitment Intelligence Platform

Introduction

Actowiz Solutions architected an end-to-end automated job scraping platform that aggregates walk-in drive postings from LinkedIn posts, company career pages, Telegram channels, consultancy websites, and public job boards — converting unstructured noise into clean, structured, deduplicated, scam-filtered recruitment intelligence in real-time.

Client Overview

The client is a fast-growing HR tech startup building India's first dedicated walk-in drive discovery platform. Walk-in interviews remain the dominant hiring channel for IT services, BPO, retail, hospitality, and entry-level engineering roles across India — yet no major job aggregator (Naukri, LinkedIn, Indeed) treats walk-ins as a first-class data type.

Walk-in opportunities are scattered across 1000+ disconnected sources: LinkedIn posts by recruiters, Telegram channels with 50K+ members, consultancy WhatsApp groups, company career sections, college placement cells, and regional job boards. Candidates were spending 4–6 hours daily manually searching across platforms. The client needed a continuous, automated, AI-enhanced pipeline to centralize this fragmented data into one searchable platform.

Industry & Geography

Industry: HR Tech / Recruitment / Job Board Aggregation

Geography: Pan-India — primary focus on Bengaluru, Hyderabad, Chennai, Pune, Mumbai, Delhi NCR, Kolkata, and Ahmedabad

Business Challenges

Highly Fragmented Source Ecosystem

Walk-in postings were spread across LinkedIn posts (unstructured text), Telegram channels (chat messages with images), consultancy career pages (varied HTML structures), company career sections (JavaScript-rendered), and small regional job boards — each requiring different scraping strategies.

Unstructured Free-Text Job Descriptions

Unlike Naukri or LinkedIn jobs which have structured fields, walk-in posts were mostly free-form text and images — "Walk-in for Java developers tomorrow at 10 AM, Marathahalli office, 2-5 yrs exp, salary upto 12 LPA" — needing intelligent parsing to extract company, role, skills, venue, date, salary, experience.

Heavy Duplication Across Sources

The same walk-in drive was often posted on 15–25 sources within hours — by recruiters, consultancies, Telegram aggregators, and the company itself. Without smart deduplication, the platform would show users the same job 25 times.

Scam & Fake Job Postings

Recruitment fraud is rampant in walk-in postings — fake offer letters, registration-fee scams, MLM disguised as jobs, and ghost openings from inactive consultancies. The platform needed automated scam detection to protect candidates.

Time-Sensitive Data Decay

Walk-in drives typically happen within 1–7 days of being posted. Stale data is useless. The pipeline needed sub-hour latency from source posting to candidate visibility.

Anti-Bot & Rate-Limiting Defenses

LinkedIn aggressively blocks scrapers, Telegram has API rate limits, and career pages use Cloudflare, hCaptcha, and JavaScript rendering as defenses.

Project Objectives

The client partnered with Actowiz Solutions to:

• Build a multi-source scraping pipeline covering 5+ source types and 200+ specific sources
• Engineer AI/NLP-based structured extraction from unstructured job text and images
• Implement intelligent duplicate detection across heterogeneous sources
• Deploy automated scam & fake job filtering using ML classification
• Ensure sub-1-hour end-to-end latency from source to clean structured database
• Architect the system for horizontal scalability to 10,000+ jobs/day
• Provide a clean REST API for the client's mobile and web apps

Actowiz Solutions Approach

Multi-Source Scraper Worker Architecture

Built independent scraper workers per source type — LinkedIn post scrapers using authenticated session rotation, Telegram channel listeners via Telethon library, headless-browser scrapers (Playwright + Puppeteer) for JavaScript-heavy career pages, and REST-based crawlers for public job boards. Each worker pushes raw payloads to a central queue.

Distributed Queue & Raw Storage

Implemented Redis-backed task queues with Celery workers, and S3-compatible object storage for raw HTML, screenshots, and Telegram message archives. Raw data is retained for 30 days for re-extraction if NLP models improve.

AI/NLP-Powered Structured Extraction

Multi-stage extraction pipeline:

• Stage 1: Regex + spaCy NER for high-confidence fields (dates, phone numbers, emails)
• Stage 2: Fine-tuned BERT classifier for job role & company name detection
• Stage 3: OpenAI GPT-4 function calling for complex fields (skills, venue, eligibility)
• Stage 4: Confidence scoring — fields below 0.7 confidence flagged for manual review

Intelligent Duplicate Detection

Multi-signal deduplication:

• Fuzzy text matching (Levenshtein distance) on company + role + date + city
• Embedding-based similarity (Sentence Transformers) for semantic duplicates
• Phone number / venue address hashing as deterministic match keys
• Time-window clustering — jobs within 48 hours and same company-role-city collapse into one canonical record with multiple source references

Scam & Fake Job Filtering

ML-based scam classifier trained on:

• Known fraud signals (registration fees, Telegram-only contact, "no experience needed" for premium roles, mismatched company emails)
• Blacklisted consultancy database
• Salary-outlier detection (₹50 LPA for fresher = flag)
• Company verification via MCA database + LinkedIn company page presence
• Honeypot detection (jobs that ask candidates to deposit money)

Key Features

• Multi-source scraping — LinkedIn, Telegram, career pages, consultancy sites, job boards
• AI/NLP-powered structured extraction with 9 critical fields per job
• Intelligent multi-signal deduplication across heterogeneous sources
• ML-based scam & fake job filtering protecting candidates from fraud
• Sub-1-hour end-to-end latency from source posting to clean database
• Horizontally scalable Celery + Kubernetes architecture
• Clean REST API (job board scraping-grade) for client app integration
• Self-healing pipeline with per-source health scoring & auto-fallbacks
• Geospatial venue search using PostGIS for "walk-ins near me" queries
• 30-day raw data retention for NLP model re-training and back-filling

Business Impact

• ₹0 manual curation cost replacing a 4-person manual aggregation team
• 22x faster candidate-to-job discovery — from 4-hour daily search to 10-minute app session
• 94.6% extraction accuracy across the top 5 structured fields
• 97.2% deduplication precision — candidates never see the same job twice
• 300–500 scam jobs filtered daily, protecting candidate trust and platform reputation
• 38-minute average source-to-app latency vs industry-standard 6–12 hours
• 15,000+ raw jobs processed daily at <$0.02 per job processing cost
• 3.8x user retention improvement after launching the automated feed vs the earlier manual model
• 400,000+ Monthly Active Users reached within 6 months of pipeline going live

Testimonial

"Walk-in jobs were impossible to aggregate — too messy, too scattered, too fast-moving. Actowiz Solutions cracked it. We went from a manual content team to an autonomous data engine in 90 days."

— Founder & CEO, Walk-In Drive Discovery Platform

Conclusion

Walk-in drives represent one of India's most fragmented yet highest-volume hiring channels — and one that traditional job aggregators have completely ignored. Actowiz Solutions engineered a category-defining recruitment intelligence pipeline that turns chaotic, multi-source, free-text job posts into clean, structured, fraud-filtered, real-time data feeds.

By combining battle-tested scraping infrastructure with modern NLP, OpenAI function calling, and intelligent deduplication, the platform now powers a candidate-facing app that serves hundreds of thousands of job-seekers across India daily — proving that even the messiest data ecosystems can be tamed with the right architecture.

FAQs

How does Actowiz scrape walk-in jobs from LinkedIn without getting blocked?

We use authenticated session rotation, intelligent request throttling, residential proxy networks, and behavioral-pattern emulation to mirror human browsing — keeping scrape success rates above 90% sustainably.

Can the system handle job postings shared as images on Telegram?

Yes. Image-based postings are processed through an OCR + vision-language model pipeline that extracts text from screenshots and infographics commonly shared in Telegram job channels.

How does deduplication work when the same job is posted by 20 different sources?

We combine fuzzy text matching, semantic embedding similarity, deterministic keys (phone number, venue address), and time-window clustering. All sources are linked to one canonical job record with multiple source references preserved.

How accurate is the AI-powered field extraction?

Across the top 5 critical fields (company, role, skills, experience, venue), our extraction pipeline maintains 94.6% accuracy in production. Fields below 0.7 confidence are flagged for human review.

Can this scraping pipeline be customized for other job categories beyond walk-ins?

Absolutely. The same architecture extends to remote jobs, contract roles, internships, executive search, and niche industry-specific recruitment intelligence. Talk to our team for custom requirements.

What technologies power the backend?

FastAPI for the REST API, PostgreSQL with PostGIS for structured storage, Redis + Celery for distributed queueing, Docker + Kubernetes for orchestration, and a multi-stage NLP pipeline combining spaCy, fine-tuned BERT, and OpenAI GPT-4 function calling.