Role Overview

We are looking for a Senior Web Scraping Engineer to design, build, and operate high-reliability web data acquisition systems in agile web environments.

The role is hands-on and technical, focused on solving real-world scraping challenges such as anti-bot defenses, session management, browser automation, and large-scale parallel crawling.

Key Responsibilities

Core Web Scraping & Data Acquisition

- Design and implement high-parallel crawlers for complex, dynamic websites
- Handle JavaScript-heavy sites using Python / Playwright / Puppeteer / CDP
- Implement robust session management, cookies, headers, and identity rotation
- Build resilient pipelines for structured & unstructured data extraction
- Ensure data accuracy, freshness, and consistency at scale

Anti-Bot & Protection Handling

- Analyze and mitigate anti-bot protections, including:
- Cloudflare
- Akamai
- Qrator
- ServicePipe
- Variti (and similar)





- Work with browser-level and network-level fingerprinting:
- TLS / JA3
- CDP fingerprints
- Header and behavior-based detection

- Integrate and optimize CAPTCHA solving strategies (where legally permitted)

Required Skills & Experience

Must-Have

- 1-3+ years of hands-on web scraping / crawling experience
- Strong proficiency in Python (Scrapy, asyncio, aiohttp, Playwright)
- Deep understanding of:
- HTTP, cookies, headers, redirects
- Browser automation & JS execution

- Experience scraping highly protected websites
- Solid debugging skills in hostile scraping environments

Nice-to-Have (Not Mandatory)

- Familiarity with:
- Kubernetes / Helm (user-level, not platform owner)
- CI/CD pipelines
- Distributed task queues (Redis, RabbitMQ, Kafka)

- Experience mentoring junior engineers

📌 Web Scraping Engineer (Pune)
🏢 Krawlnet Technologies
📍 Pune