Roles and Responsibilities :

Solution Architect – Web Scraping & Data Engineering

Job Mode: Remote
Type of Job: Full time
Experience: 10+ Years
Employment Type: Contract

Role Overview

We are looking for a Solution Architect with strong expertise in Web Scraping & Data Extraction and supporting capabilities in Data Engineering. The role will lead architecture, solution design, pre-sales activities, technical estimations, and delivery of large-scale data extraction platforms.

Key Responsibilities

Web Scraping (Primary Focus)

- Design and architect large-scale web scraping solutions.
- Build resilient crawlers for dynamic websites, anti-bot protected platforms, and large-volume data extraction.
- Define extraction, parsing, validation, deduplication, and data quality frameworks.
- Evaluate scraping frameworks, proxy solutions, and anti-bot strategies.

Pre-Sales & Solutioning

- Participate in RFP/RFI responses and proposal creation.
- Conduct technical discovery and feasibility assessments.
- Prepare effort estimations, architecture diagrams, and solution recommendations.
- Support client discussions and technical presentations.

Data Engineering (Secondary Focus)

- Design ETL/ELT pipelines and data architectures.
- Build scalable data lake, warehouse, and analytics solutions.
- Define data quality, observability, governance, and monitoring standards.

Required Skills

Web Scraping

- Python: Scrapy, BeautifulSoup, Requests, lxml
- Playwright, Selenium, Puppeteer
- API extraction, XPath, CSS Selectors,



Regex
- Proxy management and anti-bot handling
- Distributed crawling and orchestration frameworks

Data Engineering

- Airflow, Prefect, Dagster
- Spark/PySpark, Databricks
- Kafka and streaming solutions
- Snowflake, Redshift, BigQuery, Synapse
- SQL, PostgreSQL, MySQL, NoSQL databases

Cloud & DevOps

- AWS/Azure/GCP
- Docker, Kubernetes
- Terraform or Infrastructure as Code
- GitHub Actions, GitLab CI, Jenkins
- Monitoring tools such as Grafana, Datadog, ELK

Required Experience
- 10+ years in software engineering.
- 5+ years in large-scale web scraping and data extraction.
- 3+ years in data engineering and pipeline architecture.
- Experience leading architecture discussions, solutioning, and technical teams.
- Robust client-facing and pre-sales experience.

Preferred Skills
- ML/AI-assisted extraction.
- GraphQL and API reverse engineering.
- Experience in e-commerce, market intelligence, financial, travel, or real-estate domains.
- Open-source contributions in scraping or data engineering.

Key Deliverables
- Solution architecture and technical design documents.
- RFP/RFI technical proposals and estimations.
- HLD/LLD documentation.
- PoCs and reference implementations.
- Monitoring, governance, and operational frameworks.

Ideal Candidate: A senior architect who can own end-to-end scraping architecture, engage with clients during pre-sales, estimate projects, guide engineering teams, and deliver scalable data platforms.

📌 Solution Architect Scrapy & Lead Scrapy (Tamil Nadu)
🏢 Merit Data and Technology
📍 Tamil Nadu