Job Title: Web Crawling Engineer

Experience:3-5 Years

Location: Hyderabad

We are looking for a skilled Web Crawling Engineer with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting.

Key Responsibilities

- Design, develop, and maintain scalable and efficient web crawlers for data extraction.
- Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
- Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
- Implement and manage proxy rotation and session management strategies.
- Ensure high-quality and structured data extraction, cleaning, and storage.
- Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
- Collaborate with the data engineering team to integrate crawled data into pipelines.
- Stay updated with the latest tools, techniques, and best practices in web crawling.

Required Skills & Qualifications

- Strong programming skills in Python.
- Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).




- Hands-on experience solving blocking issues, handling
- Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques.
- Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.
- Experience with databases (SQL/NoSQL) for storing crawled data.
- Strong debugging, problem-solving, and analytical skills.

Valuable to Have

- Experience with cloud environments (AWS, GCP, Azure).
- Knowledge of data pipelines and ETL processes.

Education & Experience

- Bachelor’s degree in Computer Science, Information Technology, or equivalent practical experience.
- 3+ years of experience in web crawling or related fields

OSI Digital was founded in 1993, in California and now has offices in the US, Canada, India, Philippines, Dubai, and Australia. Our client base ranges from start-ups to Fortune 500 corporations, including Barnes & Noble College, Hyundai, Toyota, Gilead, Allergan, Yamaha, Planview, Globe and Sketchers. OSI Digital has developed strong partner alliances with the world’s leading technology providers such as Salesforce, Boomi, Tableau, Oracle, Microsoft, and Amazon Web Services.

📌 Data Engineer (Web Scraping/Crawling) (Hyderabad)
🏢 OSI Digital
📍 Hyderabad