:

Solution Architect - Web Scraping &

- Data Engineering

RFP Reference Document

Role Title : Solution Architect - Web Scraping (Primary) &

- Data Engineering (Secondary)

Engagement Type : Contract / RFP-based deployment

Reporting To : Program / Delivery Manager

Location : [Onsite / Hybrid / Remote - to be specified]

Primary Focus : Web Scraping &

- Data Extraction Architecture (~70%)

Secondary Focus : Data Engineering &

- Pipeline Architecture (~30%)

1.

Role Summary :

We are seeking a senior Solution Architect to lead the design and delivery of large-scale web scraping and data extraction solutions, with strong supporting expertise in data engineering. The architect will be the technical authority on all scraping initiatives - from pre-sales solutioning and effort estimation through to architecture, build, and stabilization.

The primary mandate is web scraping: designing resilient crawlers, anti-bot strategies, distributed extraction systems, and compliance frameworks.

The secondary mandate is data engineering: ensuring extracted data flows reliably into well-modelled, query-ready storage layers and downstream analytical or operational systems.

2. Pre-Sales, Proposal &

- Estimation Responsibilities :

This is a non-negotiable part of the role.

The Solution

Architect is mandatorily involved in every new scraping chance from the proposal stage onward, and is jointly accountable with the sales / delivery leadership for the technical correctness of all proposals.

Mandatory Involvement Areas :

- Author or co-author the technical solution section of every new RFP / RFI / proposal document for scraping projects.
- Lead technical discovery calls with prospective clients to understand target sources, data SLAs, volume, and compliance constraints.
- Conduct target-site feasibility assessments - anti-bot complexity, dynamic content, login walls, geo-restrictions, rate limits - and document findings before commercials are committed.
- Own end-to-end effort estimation for scraping engagements: crawler build effort, infrastructure sizing, proxy and CAPTCHA cost projections, maintenance overhead, and contingency.
- Produce solution architecture diagrams, tech stack recommendations, and assumption logs as part of every proposal.
- Define and document SLAs, KPIs, and acceptance criteria proposed to the client.
- Participate in client orals, technical defence sessions, and commercial negotiations as the technical SPOC.
- Maintain an internal estimation knowledge base - reusable estimation templates, complexity matrices, target-site classification, and historical actuals - and continuously refine it after every project closure.
- Sign off on the technical feasibility and risk profile of every proposal before submission. No scraping proposal goes out without architect approval.

3. Core Responsibilities :

3.1 Scraping Architecture &

- Design (Primary) :
- Design end-to-end scraping solutions covering crawl orchestration, extraction, parsing, storage, and downstream data consumption.
- Define reference architectures for high-volume, high-velocity, and high-variety scraping use cases.
- Architect resilient systems handling JavaScript-heavy sites, CAPTCHAs, rate limiting, IP blocking, and frequent DOM changes.
- Evaluate and select frameworks, proxy networks, and anti-bot bypass strategies aligned with cost, performance, and compliance.
- Design data quality, deduplication, validation, and schema-evolution strategies for scraped data.

3.2 Data Engineering Architecture (Secondary) :

- Design downstream data pipelines that ingest, transform, and serve scraped data to analytics, ML, or operational consumers.
- Architect lakehouse / warehouse layers, define data modelling standards (dimensional, Data Vault, or hybrid), and govern schema evolution.
- Define ELT/ETL patterns, orchestration strategy, and SLA-backed data freshness commitments.
- Establish data quality, observability, lineage, and cataloguing practices across the platform.
- Recommend storage formats, partitioning, and indexing strategies for cost and query performance.

3.3 Delivery Leadership :





- Translate business requirements into technical specifications, sprint plans, and implementation roadmaps.
- Produce HLD, LLD, and Architecture Decision Records (ADRs) for each engagement.
- Provide hands-on guidance, perform code reviews, and mentor scraping and data engineers.
- Drive proof-of-concept (PoC) builds for complex or high-risk targets before full-scale rollout.

3.4 Compliance, Risk &

- Governance :
- Ensure all scraping work adheres to applicable laws and platform terms (GDPR, CCPA, DPDP, robots.txt, copyright).
- Define and enforce ethical scraping practices, request throttling, and PII handling guidelines.
- Conduct risk assessments for each target source and recommend mitigations.

3.5 Performance, Cost &

- Operations :
- Establish SLAs for crawl freshness, completeness, and accuracy.
- Design monitoring, alerting, and self-healing mechanisms for scraping pipelines.
- Optimise infrastructure cost (compute, proxies, storage) without compromising delivery KPIs.

4.

Required Technical

Skills - Primary (Web Scraping) :

These are non-negotiable. The candidate must demonstrate deep, hands-on expertise in each of the following:

4.1 Scraping Frameworks &

- Tooling :
- Expert-level Python: Scrapy, BeautifulSoup, lxml, Requests, httpx, parsel.
- Headless browser automation: Playwright, Puppeteer, Selenium, Pyppeteer.
- Node.js scraping stack (where applicable): Puppeteer, Cheerio, Crawlee.

4.2 Anti-Bot &

- Evasion Strategy :
- Proven experience bypassing Cloudflare, Akamai Bot Manager, DataDome, PerimeterX, Imperva, Kasada.
- CAPTCHA handling: reCAPTCHA v2/v3, hCaptcha, FunCaptcha, image / audio solvers; integration with 2Captcha, Anti-Captcha, CapSolver.
- TLS / JA3 / JA4 fingerprinting awareness
- HTTP/2 fingerprint evasion; browser fingerprint spoofing.
- Stealth plugins, user-agent rotation, header normalisation, cookie / session management at scale.

4.3 Proxy &

- Network Infrastructure :
- Hands-on experience with rotating, residential, mobile, ISP, and datacenter proxies.
- Integration with providers: Bright Data, Oxylabs, Smartproxy, NetNut, IPRoyal, SOAX.
- Proxy pool design, health-checking, geo-targeting, sticky sessions, and cost optimisation.

4.4 Parsing &

- Extraction :
- XPath, CSS selectors, regex, JSON-LD, microdata, RDFa.
- Reverse-engineering of internal / mobile APIs, GraphQL endpoints, and XHR traffic.
- ML/LLM-assisted extraction for unstructured layouts (nice-to-have, increasingly expected).

4 Distributed Crawling &

- Orchestration :
- Scrapy-Redis, Scrapy Cluster, Frontera, Crawlee, or equivalent distributed crawling frameworks.
- Job scheduling and orchestration with Apache Airflow, Prefect, Dagster, or Celery.
- Queue-based architectures using Kafka, RabbitMQ, AWS SQS, GCP Pub/Sub.

Required Technical Skills - Secondary (Data Engineering) :

Working-to-strong proficiency expected. The architect must be able to design, review, and guide data engineering work without depending on a separate data architect.

5 Data Pipelines &

- Orchestration :
- ETL / ELT design patterns, idempotent pipelines, CDC (Change Data Capture), incremental loads.
- Apache Airflow, Prefect, Dagster, AWS Glue, Azure Data Factory, GCP Dataflow.
- Stream processing: Kafka Streams, Apache Flink, Spark Structured Streaming.
- Batch processing: Apache Spark (PySpark), Databricks, EMR, Dataproc.

5 Data Modelling &

- Storage :
- Dimensional modelling (Kimball), Data Vault 2.0, normalised vs. denormalised trade-offs.
- Data warehouses: Snowflake, BigQuery, Redshift, Synapse, Databricks SQL Warehouse.
- Data lakes / lakehouses: Delta Lake, Apache Iceberg, Apache Hudi on S3 / GCS / ADLS.
- OLTP databases: PostgreSQL, MySQL
- NoSQL: MongoDB, DynamoDB, Cassandra, Elasticsearch, Redis.




- File formats: Parquet, Avro, ORC, JSON, CSV; partitioning, bucketing, compaction strategies.

5 Transformation &

- Quality :
- dbt (data build tool) for transformation, testing, and documentation.
- Data quality frameworks: Great Expectations, Soda, Deequ, custom validators.
- Data lineage and cataloguing: DataHub, OpenMetadata, Amundsen, Atlan, Collibra.

5. Cloud, DevOps &

- Observability :
- Strong on at least one of AWS, GCP, Azure (compute, storage, IAM, networking, serverless).
- Containerisation and orchestration: Docker, Kubernetes (EKS / GKE / AKS), ECS.
- Infrastructure as Code: Terraform, Pulumi, CloudFormation.
- CI/CD: GitHub Actions, GitLab CI, Jenkins, Argo CD.
- Observability: Prometheus, Grafana, ELK / OpenSearch, Datadog, Sentry, OpenTelemetry.

6.

Experience &

Qualifications :

- Bachelor's or Master's degree in Computer Science, Engineering, or related field.
- 10+ years of overall software engineering experience.
- 5+ years dedicated to large-scale web scraping / data extraction (primary).
- 3+ years of hands-on data engineering experience covering pipelines, warehouses / lakehouses, and orchestration (secondary).
- Proven track record of architecting scraping platforms processing millions of pages per day across diverse target sites.
- Prior experience as Solution Architect, Tech Lead, or Principal Engineer leading teams of 5+ engineers.
- Demonstrated experience in pre-sales / proposal authoring / RFP responses for scraping or data engineering engagements.
- Client-facing consulting experience strongly preferred.

7. Soft Skills :

- Excellent written and verbal communication; able to defend technical proposals to CXO-level audiences.
- Strong commercial acumen - understands the cost / risk / quality trade-offs in estimation.
- Analytical, structured problem-solving mindset.
- Ownership-driven, comfortable being the single point of technical accountability.
- High standards for documentation, knowledge transfer, and reusability.

8.

Expected

Deliverables (RFP Scope) :

The Solution Architect deployed under this RFP shall be responsible for producing, at minimum, the following artifacts:

Pre-Sales / Proposal Phase :

- Technical solution sections of all proposal documents.
- Target-site feasibility and risk assessment reports.
- Effort estimation models, assumption logs, and complexity matrices.
- Solution architecture diagrams and tech stack recommendations for proposals.

Delivery Phase :

- High-Level Design (HLD) and Low-Level Design (LLD) documents per initiative.
- Architecture Decision Records (ADRs).
- Reference implementations / PoCs for complex extraction scenarios.
- Code review records and engineering standards documentation.
- Operational runbooks, monitoring playbooks, and incident response procedures.
- Compliance and data governance documentation.
- Knowledge transfer sessions and final handover documentation.

9. Nice-to-Have :

- ML / NLP-based extraction, entity resolution, or LLM-assisted parsing experience.
- Exposure to GraphQL, gRPC, mobile API reverse-engineering, mitmproxy / Charles workflows.
- Domain experience: e-commerce price intelligence, market research, financial data, real estate, travel aggregation, hospitality.
- Open-source contributions to scraping or data engineering ecosystems.
- Familiarity with cross-jurisdictional legal frameworks for automated data collection.
- Experience with reverse ETL tools (Hightouch, Census) and feature stores.

10.

Evaluation

Criteria (RFP Response) :

Vendors proposing candidates against this role will be evaluated on:

- Depth and recency of relevant scraping architecture experience (primary weight).
- Strength of supporting data engineering experience (secondary weight).
- Quality of past project case studies - scale, complexity, business outcomes.
- Demonstrated capability in proposal authoring and technical estimation.
- Technical depth in interviews, whiteboarding, and architecture discussions.
- Communication and stakeholder management capability.
- Commercial competitiveness of the rate card.
- Availability and ramp-up timeline.

📌 Merit Group - Solution Architect - Web Scraping & Data Engineering (India)
🏢 Merit Data and Technology
📍 India