We are looking for a Senior Web Scraping Engineer to design, build, and operate high-reliability web data acquisition systems in dynamic web environments.
The role is hands-on and technical, focused on solving real-world scraping challenges such as anti-bot defenses, session management, browser automation, and large-scale parallel crawling.
Key Responsibilities
Core Web Scraping & Data Acquisition
- Design and implement high-parallel crawlers for complex, agile websites
- Handle JavaScript-heavy sites using Python / Playwright / Puppeteer / CDP
- Implement robust session management, cookies, headers, and identity rotation
- Build resilient pipelines for structured & unstructured data extraction
- Ensure data accuracy, freshness, and consistency at scale
Anti-Bot & Protection Handling
- Analyze and mitigate anti-bot protections, including:
- Cloudflare
- Akamai
- Qrator
- ServicePipe
- Variti (and similar)
- Work with browser-level and network-level fingerprinting:
- TLS / JA3
- CDP fingerprints
- Header and behavior-based detection
- Integrate and optimize CAPTCHA solving strategies (where legally permitted)
Required Skills & Experience Must-Have
- 1-3+ years of hands-on web scraping / crawling experience
- Strong proficiency in Python (Scrapy, asyncio, aiohttp, Playwright)
- Deep understanding of
- HTTP, cookies, headers, redirects
- Browser automation & JS execution
- Experience scraping highly protected websites
- Solid debugging skills in hostile scraping environments