This role is for one of the Weekday's clients
Salary range: Rs ******* - Rs ******* (ie INR ***** LPA)
Min Experience: 5 years
Location: Pune
JobType: full-time
We are looking for an experienced
Python Lead
with deep expertise in
web scraping and data extraction
to lead the design, development, and optimization of scalable scraping solutions. This role requires strong hands-on skills with
Scrapy, Beautiful Soup (BeautifulSoup), and web scraping frameworks
, along with the ability to guide a team, define best practices, and deliver reliable data pipelines at scale.
As a Python Lead, you will take ownership of complex scraping architectures, ensure high data quality, handle anti-bot challenges, and mentor developers while working closely with product, data, and business stakeholders.
Key Responsibilities
Lead the
end-to-end design and development of web and data scraping solutions
using Python.
Architect and maintain scalable scraping systems using
Scrapy framework
,
Beautiful Soup / BeautifulSoup
, and custom Python solutions.
Handle complex scraping scenarios including
dynamic websites, pagination, authentication flows, JavaScript-rendered content, and APIs
.
Implement techniques to manage
rate limiting, IP rotation, proxy management, CAPTCHA handling, and bot detection mechanisms
.
Ensure
data accuracy, consistency, and reliability
, including validation, cleaning, and structured storage of scraped data.
Optimize scraping performance for
speed, scalability,



and fault tolerance
.
Lead code reviews, enforce
coding standards and best practices
, and ensure maintainable and testable codebases.
Mentor and guide junior developers, providing technical leadership and knowledge sharing.
Collaborate with cross-functional teams to understand data requirements and translate them into robust scraping solutions.
Ensure compliance with
legal, ethical, and website terms of service
related to web scraping activities.
Required Skills & Qualifications
5–10 years of hands-on Python development experience
, with a strong focus on
web scraping and data scraping
.
Extensive experience with
Scrapy
,
Scrapy Framework
, and
Beautiful Soup / BeautifulSoup
.
Strong understanding of
HTML, CSS, XPath, and regular expressions
.
Experience scraping
large-scale and high-volume websites
.
Knowledge of handling
dynamic content
using headless browsers or JavaScript rendering techniques.
Solid experience with
databases (SQL/NoSQL)
and data formats such as JSON, CSV, and XML.
Solid debugging, problem-solving, and performance optimization skills.
Experience leading or mentoring teams in a technical capacity.
Nice to Have
Experience with
cloud platforms
, distributed systems, or task queues.
Familiarity with
data pipelines, ETL processes
, or analytics workflows.
Exposure to monitoring, logging, and alerting for scraping systems.

📌 Python Lead (Maharashtra)
🏢 Weekday AI
📍 Maharashtra