- Role Overview: Design, develop, test, and maintain scalable web scraping solutions for competitor and market intelligence.
- Core Python: Robust hands-on expertise in Python (OOP, exception handling, regex, JSON/XML processing, modular design).
- Scrapy Framework: Deep experience building production Scrapy spiders, Item Loaders, Pipelines, Middlewares, AutoThrottle, and concurrency settings.
- Browser Automation: Use Selenium and Playwright for JavaScript-heavy sites, wait strategies, infinite scroll, pop-ups, and dynamic interactions.
- Web & API Extraction: Inspect traffic via Chrome DevTools to extract data from REST/JSON APIs, XHR/Fetch, and HTML using XPath, CSS selectors, lxml, and BeautifulSoup.
- Direct HTTP: Utilize requests/httpx with proper session, cookie, header, retry, and redirect management where browser automation is unnecessary.




- Location & Variants: Accurately map location-specific data (country, region, store, airport) and product variants (size, color, pack size, SKU, pricing).
- Data Quality & Testing: Clean and normalize extracted fields against strict schemas, validate record counts, and write unit tests using pytest.
- Anti-Bot & Maintenance: Troubleshoot frontend/API breaks and mitigate 403/429 blocking and rate limits using proxy rotation and session handling.
- DevOps & Bonus Skills: Follow Git and Linux workflows; familiarity with Docker, SQL, CI/CD, CAPTCHA solving, and Playwright network interception is a plus.
- 3-7 years of experience (Hybrid)

📌 Scraper Developer (Bengaluru)
🏢 Recruise: Executive Search and Talent Advisory
📍 Bengaluru