Job Title - Python Web Scraping Engineer
Department: Engineering / Data
Employment Type: Full-Time / Contract
Location: Remote / On-site / Hybrid
Experience Level: Mid-Level / Senior

About the Role
We are looking for a highly skilled Python Web Scraping Engineer to design, build, and maintain largescale web data extraction pipelines. In this role, you will be responsible for the end-to-end lifecycle of web scraping projects—from reverse-engineering target websites and navigating anti-bot mechanisms to cleaning, structuring, and storing the extracted data.

The ideal candidate has a deep understanding of HTTP protocols, asynchronous Python programming, headless browsers, and modern data engineering practices. You will work closely with our Data and Product teams to ensure high-quality data feeds that power our core intelligence products.

Key Responsibilities

Crawler Development: Architect, develop, and maintain robust, scalable web scrapers and crawlers using Python to extract data from a wide variety of websites and APIs.

Anti-Bot Evasion: Investigate and resolve complex crawling issues, including bypassing CAPTCHAs, handling rate limits, managing proxy rotations, and overcoming anti-scraping technologies (e.g., Cloudflare, DataDome).

Dynamic Content: Utilize headless browsers and browser automation tools to scrape JavaScript heavy and dynamic websites.

Data Pipelines: Build reliable data pipelines to clean, validate, transform, and load raw HTML/JSON data into structured databases.

Performance Optimization: Implement asynchronous and concurrent processing to maximize scraping speed while respecting target server limits.





Monitoring & Reliability: Set up alerting and logging systems to detect crawler failures, UI changes on target sites, and data quality degradation.

Job Description:
Python Web Scraping Engineer Page 1 of 2 Required Qualifications Experience: 3+ years of professional experience in software engineering, with a primary focus on web scraping, data mining, or web automation.

Programming: Advanced proficiency in Python 3.x and asynchronous programming.

Scraping Frameworks: Deep hands-on experience with libraries such as Scrapy, BeautifulSoup, and Requests.

Browser Automation: Strong experience handling dynamic rendering using Playwright, Selenium, or Puppeteer.

Web Fundamentals: Thorough understanding of DOM parsing, HTML, CSS selectors, XPath, HTTP methods, headers, and cookie/session management.

Database Management: Proficiency with relational databases (PostgreSQL, MySQL) and NoSQL databases (MongoDB, Redis) for data storage.

Preferred Qualifications
Experience with distributed task queues and message brokers (Celery, RabbitMQ, Kafka).
Familiarity with containerization and cloud orchestration (Docker, AWS, GCP).
Experience with LLM-based structured data extraction.
Knowledge of reverse-engineering mobile APIs (Android/iOS).

What We Offer
Competitive salary, performance bonuses, and equity options.
Flexible working hours and work from home options.
Professional development budget for courses and conferences.

Interested candidates can email their resume directly at [email protected]

Pay: ₹25,000.00 - ₹75,000.00 per month

Benefits:

- Flexible schedule
- Work from home

Work Location: Remote

📌 Python Web Scraping Engineer (India)
🏢 Neo BPO Solutions
📍 India