Engineering Ahmedabad / Remote Full-time as per the market standards
About The Role
We're looking for a Senior Web Crawling Engineer to help build and maintain the web crawling infrastructure that powers our data extraction and analytics platforms. You'll work on developing robust, scalable crawlers that extract valuable data from complex websites and APIs.
Key Responsibilities
Architect, design, and implement scalable web crawlers using Scrapy framework as the primary technology
Build reusable Scrapy spiders, middlewares, pipelines, extensions, and custom schedulers
Develop distributed crawling solutions using Scrapy Cluster / Redis / Kafka-based queue systems
Optimize Scrapy performance (auto-throttling, concurrency tuning, memory management)
Implement advanced request fingerprinting, retry logic, proxy rotation, and user-agent rotation
Implement strategies to bypass anti-scraping mechanisms (rate limiting, CAPTCHA handling, fingerprinting)
Work with headless browsers (Playwright/Selenium)
integrated with Scrapy for JS-heavy websites
Required Qualifications
5+ years of strong Python development experience
Extensive hands-on experience with Scrapy (core framework, pipelines, middleware, extensions)
Deep understanding of Scrapy internals (Twisted, asynchronous networking)
Strong experience with XPath, CSS selectors, JSONPath
Solid understanding of HTTP/HTTPS protocols, headers, cookies, sessions
Experience handling agile websites and API-based scraping
Nice to Have
Experience integrating Scrapy with Playwright or Selenium
Knowledge of bot detection techniques and mitigation
Experience building REST APIs for crawl orchestration
Familiarity with cloud deployment and DevOps practices
What We Offer
Competitive compensation package
Hybrid work model with flexible hours
Professional development opportunities
Team building events and activities
Generous PTO and holiday schedule
Apply for this Position
Full N
📌 Senior Web Crawling Engineer (Ahmedabad)
🏢 Hexanix One
📍 Ahmedabad
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.