- Web Scraping Competitor Onboarding - 65%
- Analyse assigned websites using Chrome DevTools (Network, Elements and Console) to determine whether data is best extracted from HTML, XHR/Fetch requests, REST/JSON APIs, embedded JSON, or browser-rendered content.
- Develop production-grade spiders using the Scrapy framework, including Spiders, Requests/Responses, Items, Item Loaders, Pipelines, Middlewares and project/settings configuration.
- Use Selenium WebDriver or Playwright for Python when required data cannot reliably be obtained through Scrapy or direct HTTP requests.
- Handle JavaScript-rendered content and browser interactions such as dropdowns, pop-ups, location/airport selection, load-more buttons and infinite scrolling.
- Implement and maintain location-specific crawls where product assortment, pricing, promotions, availability or content varies by country, region, airport, store or selected location. Handle location selection through URLs, APIs, cookies, sessions or browser interactions as required.
- Handle product variants accurately, including size, volume, pack size, colour or other product options, ensuring relevant variants and their associated prices, availability, identifiers and attributes are extracted and mapped correctly.
- Build robust XPath and CSS selectors capable of handling nested and changing HTML structures.
- Implement numbered,



next-page, load-more, infinite-scroll and API/cursor-based pagination.
- Extract structured product information such as product name, brand, category, regular/promotional price, currency, availability, product URL, image URL, SKU/product identifiers and other required attributes.
- Work with requests/httpx and direct APIs where browser automation is unnecessary; parse nested JSON and embedded JavaScript/JSON objects.
- Implement request/session management including headers, cookies, retries, timeouts, redirects, throttling and proxy configuration.
- Develop reusable scraping components and avoid duplicated website-specific logic wherever practical.
- Data Quality, Testing Validation - 20%
- Validate scraper output against the source website and defined data schema.
- Implement field validation, normalization and cleaning for prices, currencies, identifiers, availability and text fields.
- Handle missing fields, malformed responses, duplicate records and unexpected page structures gracefully.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 COE - Scrapper developers (Bengaluru)
🏢 Avolta
📍 Bengaluru