Role Overview

We are looking for a Web Scraping Engineer to design, build, and operate high-reliability web data acquisition systems in dynamic web environments. The role is hands-on and technical, focused on solving real-world scraping challenges such as anti-bot defenses, session management, browser automation, and large-scale parallel crawling.
Key Responsibilities
Core Web Scraping & Data Acquisition
Design and implement high-parallel crawlers for complex, agile websites
Handle JavaScript-heavy sites using Python / Playwright / Puppeteer / CDP
Implement robust session management, cookies, headers, and identity rotation
Build resilient pipelines for structured & unstructured data extraction
Ensure data accuracy, freshness, and consistency at scale

Anti-Bot & Protection Handling
Analyze and mitigate anti-bot protections, including:
Cloudflare
Akamai
Qrator
ServicePipe
Variti (and similar)
Work with browser-level and network-level fingerprinting:
TLS / JA3
CDP fingerprints
Header and behavior-based detection
Integrate and optimize CAPTCHA solving strategies (where legally permitted)





Required Skills & Experience

Must-Have
1–3+ years of hands-on web scraping / crawling experience
Strong proficiency in Python (Scrapy, asyncio, aiohttp, Playwright)
Deep understanding of:
HTTP, cookies, headers, redirects
Browser automation & JS execution
Experience scraping highly protected websites
Solid debugging skills in hostile scraping environments

Nice-to-Have (Not Mandatory)
Familiarity with:
Kubernetes / Helm (user-level, not platform owner)
CI/CD pipelines
Distributed task queues (Redis, RabbitMQ, Kafka)
Experience mentoring junior engineers

Role Details

Role: Data Engineer
Industry Type: IT Services & Consulting
Department: Engineering – Software & QA
Employment Type: Full Time, Permanent
Role Category: Software Development
Education
UG: B.Tech / B.E. in Any Specialization
Key Skills

Scrapy
Web Crawling
Python
Data Scraping
Data Extraction
About Company

KrawlNet T

📌 Web Scraping Engineer in Pune
🏢 Krawlnet Technologies
📍 Pune