Job Description
Role Overview
nWe are looking for a Web Scraping Engineer to design, build, and operate high-reliability web data acquisition systems in dynamic web environments. The role is hands-on and technical, focused on solving real-world scraping challenges such as anti-bot defenses, session management, browser automation, and large-scale parallel crawling.
nKey Responsibilities
n Core Web Scraping & Data Acquisition
n
- n
- Design and implement high-parallel crawlers for complex, dynamic websitesn
- Handle JavaScript-heavy sites using Python / Playwright / Puppeteer / CDPn
- Implement robust session management, cookies, headers, and identity rotationn
- Build resilient pipelines for structured & unstructured data extractionn
- Ensure data accuracy, freshness, and consistency at scalen

nAnti-Bot & Protection Handling
n
- n
- Analyze and mitigate anti-bot protections, including:n
- Cloudflaren
- Akamain
- Qratorn
- ServicePipen
- Variti (and similar)n
- Work with browser-level and network-level fingerprinting:n
- TLS / JA3n
- CDP fingerprintsn




- Header and behavior-based detectionn
- Integrate and optimize CAPTCHA solving strategies (where legally permitted)n

nRequired Skills & Experience
nMust-Have
n
- n
- 1–3+ years of hands-on web scraping / crawling experiencen
- Solid proficiency in Python (Scrapy, asyncio, aiohttp, Playwright)n
- Deep understanding of:n
- HTTP, cookies, headers, redirectsn
- Browser automation & JS executionn
- Experience scraping highly protected websitesn
- Solid debugging skills in hostile scraping environmentsn

nNice-to-Have (Not Mandatory)
n
- n
- Familiarity with:n
- Kubernetes / Helm (user-level, not platform owner)n
- CI/CD pipelinesn
- Distributed task queues (Redis, RabbitMQ, Kafka)n
- Experience mentoring junior engineersn

nRole Details
nRole: Data Engineer
nIndustry Type: IT Services & Consulting
nDepartment: Engineering – Software & QA
nEmployment Type: Full Time, Permanen

📌 Web Scraping Engineer in Pune
🏢 Krawlnet Technologies
📍 Pune