Build From Scratch | Freelance / Long-Term Collaboration | Profit-Sharing Prospect for the Right Partner

We are looking for a senior-level expert who can architect and develop a complete web data acquisition platform from scratch for our upcoming AI-Powered Market Intelligence Platform.
This is not a basic web-scraping project. We need someone who can build the underlying infrastructure for AI-directed, targeted and scalable data collection.

Core Architecture
The intended workflow is:

Business Objective → LLM/AI Planning → Data Acquisition Platform → Search/Google Advanced Search/Dork Discovery → URL/Source Discovery → Proxy Management → Scrapers/APIs → Targeted Data Extraction → Validation → Deduplication → Knowledge Base → AI/ML
We already have separate LLM/AI experts. The selected developer will work closely with them to convert AI-generated research/data requirements into executable, reliable scraping jobs.

Key Responsibilities
The expert must be capable of developing:
Complete scraping platform from scratch




Scrapy / Playwright / Selenium / HTTP-based scraping infrastructure
Proxy management layer with proxy pools, assignment, health monitoring, failover and multiple provider integration
Scraping orchestrator for job queues, scheduling, concurrency, retries, workers and monitoring
Google advanced-search / Dork-style discovery and search-result/URL extraction using appropriate and compliant mechanisms
Source/domain discovery and filtering
Modular connectors for eBay, Amazon, Shopify, Etsy, Reddit, YouTube review platforms and other sources
Targeted field-level extraction rather than unnecessary bulk scraping
Data validation, normalization and deduplication
Source/URL/query/data lineage
Structured database and data pipeline
APIs through which our LLM team can create, monitor, modify and retrieve scraping jobs
Monitoring, logging and error handling

Critical Requirement: AI-Directed Scraping
The system must allow our LLM layer to determine before scraping:
W

📌 Seeking Senior Freelance Web Scraping, Proxy (Kolkata)
🏢 Outthinc Global Communications
📍 Kolkata