Python Web Scraping Engineer (4+ Years Experience)
We are looking for an experienced
Python Web Scraping Engineer
who enjoys solving challenging problems around web data extraction, distributed systems, and automation. The ideal candidate has strong experience building reliable scraping systems at scale and is eager to explore AI-driven and agentic scraping technologies.
---
Experience
* 4+ years of professional experience as a Python Developer
* Strong experience building production-grade web scraping systems
---
Key Responsibilities
* Design, develop, and maintain scalable web scraping and data extraction pipelines.
* Build robust scrapers capable of handling dynamic websites, anti-bot protections, CAPTCHAs, and rate limiting.
* Develop high-performance distributed scraping infrastructure using Docker and Kubernetes.
* Optimize scraper performance, reliability, and fault tolerance for large-scale deployments.
* Build and maintain APIs using Python web frameworks such as FastAPI or Django.
* Develop asynchronous and distributed task processing pipelines using Celery.
* Monitor production systems using observability tools such as Grafana and Prometheus.
* Collaborate with engineering teams to build resilient, maintainable, and scalable data collection platforms.
* Continuously research and adopt recent scraping techniques, browser automation methods, and AI-powered extraction technologies.
* Learn and contribute to AI/ML and Agentic AI initiatives for intelligent web automation.
---
Required Skills
* Strong Python programming skills with solid understanding of:
* Object-Oriented Programming (OOP)
* Python best practices
* Clean code principles
* Extensive experience with:
* Selenium
* Undetected ChromeDriver (UC)
* Requests
* CAPTCHA solving techniques
* Browser automation
* Strong understanding of modern anti-bot mechanisms and bypass strategies.