Back to All Jobs Senior Web Crawling Engineer Engineering Ahmedabad / Remote Full-time as per the market standards About The Role We're looking for a Senior Web Crawling Engineer to help build and maintain the web crawling infrastructure that powers our data extraction and analytics platforms. You'll work on developing robust, scalable crawlers that extract valuable data from complex websites and APIs.

Key Responsibilities Architect, design, and implement scalable web crawlers using Scrapy framework as the primary technology

Build reusable Scrapy spiders, middlewares, pipelines, extensions, and custom schedulers

Develop distributed crawling solutions using Scrapy Cluster / Redis / Kafka-based queue systems

Optimize Scrapy performance (auto-throttling, concurrency tuning, memory management)

Implement advanced request fingerprinting, retry logic, proxy rotation, and user-agent rotation

Implement strategies to bypass anti-scraping mechanisms (rate limiting, CAPTCHA handling, fingerprinting)

Work with headless browsers (Playwright/Selenium)



integrated with Scrapy for JS-heavy websites Required Qualifications 5+ years of strong Python development experience

Extensive hands-on experience with Scrapy (core framework, pipelines, middleware, extensions)

Deep understanding of Scrapy internals (Twisted, asynchronous networking)

Strong experience with XPath, CSS selectors, JSONPath

Solid understanding of HTTP/HTTPS protocols, headers, cookies, sessions

Experience handling dynamic websites and API-based scraping Nice to Have Experience integrating Scrapy with Playwright or Selenium

Knowledge of bot detection techniques and mitigation

Experience building REST APIs for crawl orchestration

Familiarity with cloud deployment and DevOps practices What We Offer Competitive compensation package Hybrid work model with flexible hours Career development opportunities Team building events and activities Generous PTO and holiday schedule Apply for this Position Full Name Email Phone LinkedIn Resume PDF, DOC, or DOCX (max 5MB) Cover Letter

📌 Senior Web Crawling Engineer (Ahmedabad)
🏢 Hexanix One
📍 Ahmedabad