Python & JavaScript Developer – AI & Web ScrapingExperience:4-7 YearsLocation:RemoteMode of Engagement:Full-timeNo of Positions:3Educational Qualifications:Bachelor's degree in computer science, Information TechnologyIndustry:IT / Software DevelopmentNotice Period:Immediate Joiners PreferredAbout the RoleWe are looking for ahands-on Web Scraping / Crawling Engineerwith3–5 years of experiencein web scraping, browser automation, and scalable data extraction.The ideal candidate should have strong experience working withdynamic and JavaScript-heavy websites, building reliable crawling workflows, handling crawling failures, and working with distributed processing systems.This is a highly technical, hands-on role. You will be expected todesign, develop, debug, optimize, and maintain web crawling and data extraction systems .ResponsibilitiesDesign, develop, and maintainscalable web crawling and scraping systemsfor agile and JavaScript-heavy websites.Develop browser automation workflows usingPlaywright, Selenium, Puppeteer, or similar frameworks .Investigate and resolve crawling issues such as403/429 responses, redirects, timeouts, rendering failures, session issues, and anti-bot challenges .Develop robustretry, fallback, and failure-handling mechanismsto improve crawler reliability.Work withcookies, sessions, browser contexts, headers, proxies, and related crawling mechanismsto maintain state and improve crawling reliability.Build and optimizeconcurrent and distributed scraping workflowsusing asynchronous processing, queues, and worker-based architectures.Design data pipelines coveringURL processing, crawling, extraction, validation, transformation, and storage .Optimize crawler performance,



includingconcurrency, browser lifecycle, resource utilization, timeouts, and request handling .Implement monitoring and observability forcrawl success rates, failure types, latency, retries, and worker performance .Debug complex crawling problems and identifyroot causes rather than relying only on one-off fixes .Develop reusable crawling components and frameworks that can support multiple websites and use cases.Collaborate with data engineering, AI, backend, and product teams to deliver reliable and structured web data.Evaluate and adopt newweb crawling, browser automation, and data extraction technologieswhere appropriate.Required Skills3–5 years of hands-on experiencein web scraping, web crawling, browser automation, or web data engineering.Strong programming experience inPython. JavaScript/Node.Js is a plus.Strong hands-on experience with Scrapy,Playwright, Selenium, Puppeteer, or similar browser automation frameworks .Strong understanding ofHTTP, HTML, DOM, JavaScript rendering, redirects, cookies, sessions, headers, and browser contexts .Experience withbrowser fingerprinting, WAFs, and modern anti-bot mechanisms .Experience working withdynamic, JavaScript-heavy, and asynchronous websites .Good understanding ofasynchronous programming, concurrency,



and parallel processing .Experience withjob queues, distributed workers, or message-based processing systemssuch as Redis, RabbitMQ, Kafka, Celery, or similar technologies.Understanding ofretry mechanisms, error handling, idempotency, rate limiting, and backpressurein distributed systems.Hands-on experience withproxy management, session handling, and anti-bot challenges .Strong debugging and analytical skills, with the ability to investigate and resolve complex crawling failures.Experience working withSQL databasessuch as PostgreSQL or MySQL.Familiarity withDocker and cloud platformssuch as AWS, GCP, or Azure is a plus.Stronghands-on engineering mindsetrather than purely project or delivery management experience.Ability to independentlydebug, investigate, and solve difficult crawling problems .Strong understanding of how browsers, HTTP requests, sessions, and websites interact.Ability to design systems that arereliable, scalable, and fault tolerant .Good understanding of how to distribute and coordinate large-scale scraping workloads.Ability to identify the root cause of crawling failures and developgeneralized, reusable solutions .Willingness to work acrossscraping, backend services, distributed processing, data pipelines, and infrastructurewhen required.Strong problem-solving skills and curiosity to understand how websites behave rather than relying solely on existing scraping tools.EducationBachelor's degree in Computer Science, Information Technology, Engineering, or a related field is preferred.Equivalent practical experience in software engineering or web scraping will also be considered.

📌 Web Scraping Engineer-Python (Remote) (Maharashtra)
🏢 AIMLEAP
📍 Maharashtra