Job Title: Web Crawling Engineer

Experience:
3-5 Years

Location:
Hyderabad We are looking for a skilled Web Crawling Engineer with robust expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting.

Key Responsibilities
• Design, develop, and maintain scalable and efficient web crawlers for data extraction.
• Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
• Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
• Implement and manage proxy rotation and session management strategies.
• Ensure high-quality and structured data extraction, cleaning, and storage.
• Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
• Collaborate with the data engineering team to integrate crawled data into pipelines.




• Stay updated with the latest tools, techniques, and best practices in web crawling. Required Skills & Qualifications
• Strong programming skills in Python.
• Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).
• Hands-on experience solving blocking issues, handling
• Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques.
• Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.
• Experience with databases (SQL/NoSQL) for storing crawled data.
• Strong debugging, problem-solving, and analytical skills. Good to Have
• Experience with cloud environments (AWS, GCP, Azure).
• Knowledge of data pipelines and ETL processes. Education & Experience
• Bachelor’s degree in Computer Science, Information Technology, or equivalent practical experience.
• 3+ years of experience in web crawling or related fields

📌 Data Engineer (Hyderabad)
🏢 OSI Digital
📍 Hyderabad