Staff Engineer, Customer Data Platform

Were seeking a hands-on, architect-level Data Engineer to take technical ownership of the customer data platform (CDP) supporting our marketing, analytics, and AI-driven applications. Youll design and build the pipelines that turn customer records, behavioral events, and campaign activity into reliable, usable data for segmentation, personalization, and campaign automation.

This role combines strong Python engineering with deep expertise in data processing, lakehouse architecture, and production reliability. Youll own the platform’s ingestion, transformation, schema management, data quality, and serving capabilities, working primarily with Python, Polars, Delta Lake, PostgreSQL, and Azure.

Key Responsibilities

- Own and evolve the CDP’s data engineering architecture, from source ingestion through curated datasets and downstream consumption.
- Design, build, and optimize batch and event-driven pipelines for customer records, behavioral events, and communication activity.
- Develop efficient transformations using Polars and Delta Lake, with Spark/Databricks where appropriate for larger workloads.
- Implement schema-driven ingestion, field mapping, type normalization, validation, deduplication, and incremental upserts.
- Evolve Bronze, Silver, and Gold data layers, including schema evolution, partitioning, and reliable merge strategies.
- Build data profiling and quality controls that surface missing, invalid, inconsistent, or stale data.
- Develop datasets and processing workflows for customer segmentation, audience materialization, personalization, and analytics.




- Build and maintain FastAPI services for ingestion orchestration, schema management, job tracking, and data access.
- Engineer reliable background workers and messaging workflows, including retries, idempotency, failure recovery, and dead-letter handling.
- Optimize pipeline throughput, memory usage, query performance, and cloud operating costs.
- Own pipeline observability, automated validation, and production troubleshooting while maintaining tenant isolation and secure handling of customer data.
- Set engineering standards, mentor engineers, and collaborate with product, application, and ML teams on data contracts and platform capabilities.

Required Qualifications:
- 4+ years of professional Python experience, including substantial ownership of production data pipelines or data platforms.
- Demonstrated ability to lead architecture decisions and deliver complex data engineering initiatives from design through production operation.
- Strong experience building ETL/ELT pipelines, including incremental processing, deduplication, schema evolution, and recovery from partial failures.
- Hands-on experience with Polars, Spark/PySpark, or a comparable data processing framework,



and the ability to optimize processing for memory and throughput.
- Experience with Delta Lake or a comparable lakehouse table format, including transactional writes, merges, and schema management.
- Strong SQL and PostgreSQL skills, including data modeling, indexing, query optimization, and semi-structured data.
- Experience building Python APIs and asynchronous background services, preferably with FastAPI.
- Experience with message queues or event streaming systems, such as Azure Service Bus, Azure Event Hubs, or comparable services.
- Practical experience with cloud object storage and cloud data infrastructure. Azure experience is preferred; strong AWS or GCP experience will be considered.
- Strong understanding of data quality, automated testing, monitoring, and production incident diagnosis.
- Experience designing secure, multi-tenant systems with appropriate access controls and protection of sensitive customer data.
- Familiarity with Docker, GitHub Actions, and up-to-date CI/CD workflows.

Preferred Qualifications:
- Experience building customer data platforms or systems for customer profiles, behavioral events, segmentation, or marketing automation.
- Hands-on experience with Azure Data Lake Storage Gen2, Azure Service Bus, Azure Event Hubs, and Databricks.
- Experience using Polars lazy execution, PyArrow, Parquet, and delta-rs to process datasets efficiently.
- Familiarity with customer identity resolution, consent and preference data, and data governance.
- Experience preparing reliable datasets for downstream ML and AI applications.

📌 Python Engineer (Kolkata)
🏢 Experiture Global
📍 Kolkata