Were Hiring: Python Web Scraping & Crawling Developers

(Multiple Openings | 4 - 10 Yrs Exp | Multiple Levels)

Location: Pune / Mumbai / Bengaluru / Hyderabad / Gurugram / Chandigarh / Chennai

- Specific location and work model may vary by positions and roles.

Industry: Fortune 500 Client Projects (Staffing via Hatchtra)

Employment Type: Permanent | Contract

About the Role

We are looking for Python Web Scraping & Crawling Developers (4 - 10 years) to design, develop, and maintain high-performance web scraping, crawling, and data extraction solutions.

You will leverage your expertise in Python, BeautifulSoup, Scrapy, Selenium, Playwright, Requests, APIs, HTML/XML parsing, and data processing to build scalable and reliable web data extraction pipelines.

The role involves working with large-scale websites, dynamic web applications, structured and unstructured data, and complex crawling requirements while ensuring data quality, accuracy, and system performance.

Open Positions & Designations

- Python Web Scraping Developer (4 - 6 Yrs)
- Senior Python Scraping Engineer (6 - 8 Yrs)
- Lead Web Scraping / Crawling Engineer (8 - 10 Yrs)
- Technical Specialist Web Data Extraction (8 - 10 Yrs)

Key Responsibilities (Scale with Level) Python Web Scraping Developer (4 - 6 Yrs)

- Develop and maintain web scraping solutions using Python.
- Build web crawlers using Scrapy, BeautifulSoup, Requests, Selenium, or Playwright.
- Extract structured and unstructured data from websites and web applications.
- Handle HTML, XML, JSON, and other web data formats.
- Develop reusable scraping components and data extraction modules.
- Work with REST APIs and third-party data sources.
- Implement data cleaning, validation, transformation, and normalization.




- Troubleshoot scraping failures and maintain existing crawlers.
- Store extracted data in relational and NoSQL databases.

Senior Python Scraping Engineer (6 - 8 Yrs)

- Design and develop scalable web crawling and scraping frameworks.
- Handle dynamic websites, JavaScript-rendered content, pagination, sessions, cookies, and authentication.
- Develop robust solutions for large-scale data extraction.
- Optimize crawler performance, concurrency, memory usage, and execution time.
- Implement scheduling, monitoring, logging, and error-handling mechanisms.
- Work with APIs, proxies, headers, user agents, and rate-limiting requirements where appropriate.
- Build data pipelines to process and deliver extracted information.
- Integrate scraping solutions with databases, cloud platforms, and downstream applications.
- Conduct code reviews and mentor junior developers.

Lead Web Scraping / Crawling Engineer (8 - 10 Yrs)

- Architect large-scale web scraping and crawling platforms.
- Define technical approaches for complex and high-volume data extraction requirements.
- Design distributed crawling and scraping systems for scalability and reliability.
- Establish coding standards, reusable frameworks, and engineering best practices.
- Lead performance optimization and crawler reliability initiatives.
- Design monitoring and alerting mechanisms for crawler health and data quality.




- Collaborate with product, data engineering, QA, and business teams.
- Mentor engineering teams and provide technical guidance.
- Evaluate new technologies and approaches for web data extraction and automation.

Skills & Tools

- Programming: Python, OOP, Data Structures & Algorithms
- Scraping & Crawling: Scrapy, BeautifulSoup, Requests
- Browser Automation: Selenium, Playwright
- Web Technologies: HTML, CSS, JavaScript, XML, JSON
- APIs: REST APIs, GraphQL, API Integration
- Data Processing: Pandas, NumPy
- Databases: PostgreSQL, MySQL, MongoDB, Redis
- Cloud: AWS, Azure, GCP
- DevOps: Docker, Kubernetes, Jenkins, GitHub Actions, CI/CD
- Messaging: Kafka, RabbitMQ good to have
- Monitoring: Logging, Alerting, Performance Monitoring
- Version Control: Git, GitHub, GitLab, Bitbucket
- Testing: PyTest, UnitTest, API Testing
- Data Formats: CSV, Excel, JSON, XML, Parquet

Qualifications

- Bachelor’s/Master’s degree in Computer Science, Engineering, or a related field.
- 4+ years of professional experience in Python development with strong web scraping/crawling experience.
- Strong hands-on experience with Scrapy, BeautifulSoup, Selenium, Playwright, or Requests.
- Experience building scalable and reliable web crawlers and data extraction pipelines.
- Solid understanding of HTML, CSS, JavaScript, HTTP, REST APIs, JSON, and XML.
- Experience handling dynamic websites and JavaScript-rendered content.
- Experience with relational and/or NoSQL databases.
- Strong debugging, troubleshooting, and performance optimization skills.
- Cloud and DevOps experience is preferred.
- Experience with large-scale crawling, distributed systems, or data engineering is an advantage.

📌 Python Web Scraping & Crawling Developer (Bengaluru)
🏢 Hatchtra Innotech
📍 Bengaluru