Job Description
We are looking for a highly skilled Backend Engineer with expertise in web crawling, web scraping, and data extraction to join our team. The ideal candidate will design, develop, and maintain large-scale web crawlers and automated data collection systems while ensuring efficiency, scalability, and reliability. Responsibilities: - Design and implement scalable web crawlers to extract data from various websites and APIs. - Develop robust data parsing and cleaning pipelines. - Implement mechanisms to handle anti-scraping measures (e. g., CAPTCHA, IP blocking, User-Agent rotation, proxies, etc. ). - Optimize crawling efficiency and data storage for large-scale data extraction. - Ensure compliance with legal and ethical guidelines related to web scraping. - Work with databases (SQL/NoSQL) to store and retrieve extracted data efficiently. - Implement distributed crawling techniques using cloud services (AWS, GCP, Azure). - Maintain documentation and improve code quality through best practices and code reviews. Requirements: - 3+ years of experience in backend development,



with a focus on web crawling and web scraping. - Solid proficiency in Python (Scrapy, BeautifulSoup, Requests, Selenium, Playwright, etc. ). - Experience with Java-based crawling frameworks (e. g., Apache Nutch, Jsoup, StormCrawler) is a plus. - Solid understanding of HTTP protocols, web technologies, and browser behavior. - Experience handling headless browsers and automated scraping mechanisms. - Familiarity with distributed systems and large-scale data extraction. - Strong knowledge of database technologies (SQL, NoSQL, Elasticsearch, MongoDB, Redis, etc. ). - Experience working with asynchronous programming (asyncio, aiohttp, Celery, etc. ). - Knowledge of cloud platforms (AWS Lambda, S3 EC2 GCP, etc. ) for scalable crawling solutions. - Experience with Machine Learning/NLP for data processing and text extraction. - Knowledge of containerization and orchestration tools like Docker &

📌 Tech Associate (Bengaluru East)
🏢 HuntingCube
📍 Bengaluru East