Key Responsibilities:
Develop and maintain scalable web scraping solutions to extract data from various websites.
Optimize scraping scripts for performance, efficiency, and reliability.
Ensure compliance with legal and ethical standards for web scraping.
Work with large datasets, including data cleaning and transformation.
Collaborate with the team to integrate scraped data into databases or other storage solutions.
Troubleshoot and resolve scraping-related challenges, including CAPTCHA handling and IP blocking.
Write well-structured, maintainable, and reusable code.
Requirements:
4+ years of Python development experience.
Robust experience with web scraping frameworks like Scrapy, BeautifulSoup, Selenium, or Playwright.
Positive understanding of HTML, CSS, JavaScript, and browser automation.
Experience working with APIs (RESTful, GraphQL) and data storage solutions (SQL, NoSQL).
Familiarity with cloud-based services like AWS, Azure, or GCP is a plus.
Robust problem-solving skills and ability to work independently.