Back to All Jobs

Senior Web Crawling Engineer

Engineering Ahmedabad / Remote Full-time as per the market standards

About The Role

We're looking for a Senior Web Crawling Engineer to help build and maintain the web crawling infrastructure that powers our data extraction and analytics platforms. You'll work on developing robust, scalable crawlers that extract valuable data from complex websites and APIs.

Key Responsibilities

- Architect, design, and implement scalable web crawlers using Scrapy framework as the primary technology

- Build reusable Scrapy spiders, middlewares, pipelines, extensions, and custom schedulers

- Develop distributed crawling solutions using Scrapy Cluster / Redis / Kafka-based queue systems

- Optimize Scrapy performance (auto-throttling, concurrency tuning, memory management)

- Implement advanced request fingerprinting, retry logic, proxy rotation, and user-agent rotation

- Implement strategies to bypass anti-scraping mechanisms (rate limiting, CAPTCHA handling, fingerprinting)

- Work with headless browsers (Playwright/Selenium) integrated with Scrapy for JS-heavy websites

Required Qualifications





- 5+ years of strong Python development experience

- Extensive hands-on experience with Scrapy (core framework, pipelines, middleware, extensions)

- Deep understanding of Scrapy internals (Twisted, asynchronous networking)

- Strong experience with XPath, CSS selectors, JSONPath

- Solid understanding of HTTP/HTTPS protocols, headers, cookies, sessions

- Experience handling dynamic websites and API-based scraping

Nice to Have

- Experience integrating Scrapy with Playwright or Selenium

- Knowledge of bot detection techniques and mitigation

- Experience building REST APIs for crawl orchestration

- Familiarity with cloud deployment and DevOps practices

What We Offer

Competitive compensation package

Hybrid work model with flexible hours

Career development opportunities

Team building events and activities

Generous PTO and holiday schedule

Apply for this Position

Full Name

Email

Phone

LinkedIn

Resume

PDF, DOC, or DOCX (max 5MB)

Cover Letter

📌 Senior Web Crawling Engineer (Ahmedabad)
🏢 Hexanix One
📍 Ahmedabad