Experience : 5.00 + years

Salary : Confidential (based on experience)

Shift : (GMT+05:30) Asia/Kolkata (IST)

Opportunity Type : Hybrid ()

Placement Type : Full time Permanent Position

**(*Note: This is a requirement for one of Uplers' client - 1digitalstack.ai)

**What do you need for this opportunity?

Must have skills required: Prometheus, Grafana, opentelemetry, Datadog, RabbitMQ, Kafka, Airflow, Temporal, Prefect, Kestra

1digitalstack.ai is Looking for:**** : Data Platform Engineer Location: Gurugram

Employment Type: Full-Time — Hybrid

Experience: 5 to 8 years

Role Description We are hiring a senior individual contributor to help scale the platform behind our first-party (1P) marketplace data operations. We run high-volume authenticated crawls across 220+ global marketplaces.

As that footprint grows, the engineering challenge shifts from writing individual crawlers to building the platform that runs thousands of them predictably. This role sits at that layer. You will work on orchestration, data quality, observability, and service integration — the systems that let a large crawler fleet run at a known standard rather than case by case.

It is a hands-on engineering role with real architectural ownership.

What You Will Own

- Orchestration and job reliability
- Own the scheduling and orchestration layer that coordinates 1P crawl jobs across marketplaces, accounts, and regions.
- Design retry, backoff, dead-letter, and idempotent rerun semantics so repeated execution is always safe.
- Build heartbeat, timeout, and automated recovery patterns into long-running distributed jobs.
- Manage concurrency, rate limiting, session lifecycle, and credential rotation for authenticated portals.
- Define completion SLAs by job type and design the platform to meet them.

- Data integrity
- Design schema contracts and validation gates that data passes through before publication.
- Build deduplication into the pipeline using natural keys, content hashing, and idempotency keys.
- Implement completeness checks across marketplace, account, and date dimensions.
- Apply anomaly detection to volume, field-fill rates, and value distributions.
- Design quarantine and replay paths so questionable batches are held and reprocessed cleanly.

- Observability
- Instrument the platform with structured logging, metrics,



and distributed tracing.
- Propagate correlation identifiers across job, task, request, and output record, so any row can be traced to its source fetch.
- Design artifact capture and retention so runs are reproducible and auditable.
- Build dashboards, alerting, and runbooks that scale with the fleet.
- Treat time-to-diagnosis as a tracked engineering metric.

- Service integration
- Integrate the pipeline with authentication, relational, and analytical query services.
- Apply timeouts, circuit breakers, bulkheads, and graceful degradation across service boundaries.
- Design health checks and readiness signals that make system state explicit.
- Build for partial-dependency conditions, so the platform degrades predictably rather than unevenly.

Requirements

- 5 to 8 years building production Python systems, including async work with asyncio or equivalent.
- Deep hands-on experience with Scrapy and Playwright or Selenium at scale.
- Proven work on authenticated portals. Session handling, cookie and token lifecycle, proxy rotation, and anti-bot mitigation.
- Strong distributed queueing experience with RabbitMQ or Kafka. Consumer groups, redelivery, ordering, and dead-letter queues.
- Hands-on workflow orchestration experience. Airflow, Temporal, Prefect, Kestra, or a comparable system.
- Production experience with MongoDB and PostgreSQL, including index design and query tuning on large datasets.
- Practical observability experience. OpenTelemetry, Prometheus and Grafana or similar, structured logging, and distributed tracing.
- Solid grounding in HTTP and HTTPS, HTML, DOM, XPath, and CSS selectors.
- Comfortable with Linux, Docker, and CI/CD pipelines.
- Able to reason about a distributed failure across multiple services independently.

Nice to Have

- Trino, Presto, or comparable distributed query engines.
- Experience with e-commerce vendor and seller portals.




- Data quality tooling such as Great Expectations or Soda, or a custom equivalent.
- Infrastructure as code.
- Failure injection or chaos testing practice.
- Experience mentoring engineers through code review and design review.

What We Are Looking For

- Ownership. You take responsibility for outcomes across system boundaries.
- Systems thinking. You look for the class of problem, not just the instance.
- Curiosity to analyse and reverse-engineer websites and their defences.
- Evidence-driven engineering. You reach for logs, traces, and data first.
- Transparent written communication with data, analytics, and client-facing teams.
- Comfort in a fast-paced environment with shifting marketplace behaviour.

Company Description 1DigitalStack.ai is a Technology and Data Science product company helping brands win and grow profitably on e-commerce marketplaces across the globe. Our platforms give customers deep e-marketplace data, advanced and custom analytics, actionable intelligence, and end-to-end media optimization and automation. Brand Managers, P&L; Owners, E-commerce Managers, Channel and Category Managers, and Marketing Leaders use our solutions to unlock new revenue opportunities every day. We partner with some of India's largest consumer brands, including Unilever, Marico, Coke, Unicharm, Tata Consumer, and Dabur. We operate across 220+ global marketplaces spanning Southeast Asia, Europe, and the UAE. How to apply for this opportunity?

- Step 1: Click On Apply! And Register or Login on our portal.
- Step 2: Complete the Screening Form & Upload updated Resume
- Step 3: Increase your chances to get shortlisted & meet the client for the Interview!

About Uplers: Our goal is to make hiring reliable, simple, and fast. Our role will be to help all our talents find and apply for relevant contractual onsite opportunities and progress in their career. We will support any grievances or challenges you may face during the engagement. (Note: There are many more opportunities apart from this on the portal. Depending on the assessments you clear, you can apply for them as well).

So, if you are ready for a new challenge, a great work environment, and an opportunity to take your career to the next level, don't hesitate to apply today. We are waiting for you!

📌 Data Platform Engineer (Gurugram)
🏢 Uplers
📍 Gurugram