Solution Architect Web Scraping & Data Engineering (RFP Role)

Role Overview

Senior Solution Architect responsible for designing and delivering large-scale web scraping solutions (70%) with supporting data engineering architecture (30%). Acts as the technical authority from pre-sales to delivery.

Key Responsibilities

1. Pre-Sales & Proposal (Mandatory)

- Co-author technical sections of RFP/RFI proposals
- Lead discovery calls & assess target-site feasibility (anti-bot, login, geo restrictions)
- Own effort estimation (build, infra, proxies, CAPTCHA, maintenance)
- Create architecture diagrams, SLAs, KPIs, assumptions
- Participate in client discussions and sign off on technical feasibility
- Maintain estimation frameworks and knowledge base

2. Scraping Architecture (Primary)

- Design scalable scraping systems (crawl extract parse store)
- Handle dynamic sites, CAPTCHA, rate limits, IP blocking
- Define anti-bot strategies and proxy setups
- Ensure data quality, validation, and schema evolution

3. Data Engineering (Secondary)

- Design pipelines for ingestion, transformation, and analytics
- Define data models (dimensional/Data Vault)
- Build ETL/ELT workflows with orchestration and SLA tracking
- Implement data quality, lineage, and observability

4. Delivery Leadership

- Create HLD, LLD, ADRs
- Guide teams, review code, and mentor engineers
- Lead PoCs for complex use cases

5. Compliance & Operations

- Ensure legal compliance (GDPR, etc.) and ethical scraping




- Define monitoring, alerting, and cost optimization strategies
- Establish SLAs (freshness, accuracy, completeness)

Required Skills

Web Scraping (Primary)

- Python (Scrapy, BeautifulSoup), Playwright/Selenium, Puppeteer
- Anti-bot strategies (Cloudflare, DataDome, CAPTCHA handling)
- Proxy management (residential/mobile/datacenter)
- Parsing (XPath, APIs, GraphQL)
- Distributed crawling & orchestration (Airflow, Kafka, etc.)

Data Engineering (Secondary)

- ETL/ELT, Airflow/ADF/Glue, Spark
- Warehouses (Snowflake, BigQuery) & lakehouses (Delta, Iceberg)
- Data modelling, dbt, data quality tools
- Cloud (AWS/GCP/Azure), Docker, Kubernetes, CI/CD, observability

Experience

- 10+ years overall; 5+ years web scraping; 3+ years data engineering
- Experience designing high-scale scraping systems
- Prior Solution Architect / Tech Lead experience
- Solid pre-sales & client-facing experience

Deliverables

- Proposal solutions, estimation models, feasibility reports
- HLD, LLD, ADRs, PoCs
- Runbooks, monitoring playbooks, governance documents
- Knowledge transfer & documentation

Nice to Have

- ML/LLM-based extraction
- API reverse engineering
- Domain experience (e-commerce, finance, travel, etc.)

Evaluation Criteria

- Scraping expertise (primary focus)
- Data engineering capability
- Proposal/estimation experience
- Project scale & complexity
- Communication & commercial competitiveness

📌 Solution architect Web Scraping & Data Engineering (Tamil Nadu)
🏢 Merit Data and Technology
📍 Tamil Nadu