KEY RESPONSIBILITIES

- Design & implement governed ingestion pipelines consuming on-site & off-site events into canonical

event schema

- Build and maintain Kafka or Kinesis-based streaming pipelines with schema validation, data quality

checks, and alerting on source failures

- Integrate batch ingestion from databases and alongside the streaming path, managing orchestration

via Airflow or equivalent

- Enforce privacy flags at ingestion time quarantining or anonymising events without valid consent

before they reach downstream layers

- Maintain a raw event store with partitions by date and source, serving as the audit source for the

pipeline

- Implement deterministic identity across fragmented systems

- Build and maintain the cookie-to-login stitching mechanism

- Design identity logic spanning on-site sessions, CRM records etc

- Monitor identity resolution quality match rate, false positive rate, unresolved session ratio — and

iterate on matching logic to improve coverage over time

- Compute windowed aggregations over the event store

- Build and maintain governed, versioned feature definitions in a feature store ensuring normalisation,

encoding, and embedding lookup logic is consistent between training and serving pipelines

- Collaborate with data scientists to translate model feature requirements into production-grade

pipeline implementations with no training-serving skew

- Implement automated data contract validation between pipeline layers, schema compatibility checks,

completeness assertions, and anomaly detection

- Define and enforce SLAs on pipeline freshness, completeness, and accuracy with monitoring

dashboards and escalation paths for breaches

- Ensure PII is classified, tagged, and handled according to GDPR and CCPA requirements at every stage

of the pipeline like ingestion, storage, feature compute, and serving

- Contribute and maintain data lineage document, full traceability from raw event to feature to model

prediction

- Work closely with the Data Architect to implement data contracts and schema agreements between

pipeline layers, and flag design risks early

- Partner with ML Engineers to ensure feature pipelines meet model training and online inference

requirements





- Participate in code reviews, contribute to engineering standards, and mentor junior engineers where

applicable

REQUIRED QUALIFICATIONS

- 6-10 years of hands-on data engineering experience, with at least 2 years in a senior or lead capacity on

production systems

- Demonstrable experience building and operating event-driven, streaming data pipelines at scale in a

production environment

- Prior experience on a personalisation, recommendation, or user behavioural analytics platform is

strongly preferred

- Experience working within a regulated data environment like GDPR, CCPA, or equivalent.

- Batch and structured streaming, including windowed aggregations, stateful processing, and

performance tuning

- Event streaming platforms including producer/consumer design, partition management, and exactlyonce semantics

- Data Lakehouse engineering like Delta Lake, Apache Iceberg, or equivalent, including ACID

transactions, schema evolution, and time travel

- Pipeline orchestration using Apache Airflow, AWS Glue, Azure Data Factory, or Databricks Workflows

including DAG design, dependency management, and failure handling

- SQL and Python at production standard quality with clean, tested, version-controlled code

- Cloud data platform exposure on at least one of: AWS (S3, Glue, Kinesis, Redshift), Azure (ADLS, Data

Factory, Synapse), or Databricks

- Feature store design and operation: Databricks Feature Store, Feast, Tecton, or equivalent

- Data quality frameworks: Great Expectations, dbt tests, or equivalent for automated pipeline

validation

- Deterministic and probabilistic matching, entity deduplication, graph-based stitching

- CI/CD for data pipelines with automated testing, deployment, and monitoring using GitHub Actions,

Azure DevOps, or equivalent

PREFERRED QUALIFICATIONS





- Experience with graph data modelling and graph processing frameworks — Neo4j, Amazon Neptune,

GraphX, or GraphFrames

- Familiarity with vector embedding pipelines - batch encoding of text at scale, embedding storage, and

integration with ANN search infrastructure

- Working knowledge of NLP pipeline engineering — tokenisation, embedding generation, and chunking

for unstructured text at scale

- Experience with real-time feature serving and integration like Redis, DynamoDB, or equivalent cache

stores

- Familiarity with MLflow or equivalent model registry — understanding of how feature pipelines

connect to model training and deployment workflows

- IaC experience like Terraform, Bicep, or AWS CDK for reproducible data infrastructure deployment

- Experience with data mesh or federated data architecture patterns in multi-team environments

- Solid written and verbal communication — able to explain complex pipeline design decisions to nonengineering stakeholders clearly

- Able to make design decisions where requirements are incomplete and flag risks proactively

- Collaborative working style with cross-functional teams spanning data engineering, ML, platform, and

product

- Understanding that interfaces between pipeline layers are as important as the implementations within

them

- Takes responsibility for pipeline reliability, DQ, and SLA adherence end-to-end, not just the code

written

WHAT WE OFFER

- Awesome Culture: Creative Synergies has a flat organization and an agile culture of positivity,

entrepreneurial spirit, customer centricity, celebrating technical excellence, teamwork, and meritocracy

- Opportunity to work with Customers who are technology Leaders (including Global Fortune 500

Customers) & work on Real-World Problems that matter and are often mission-critical

- Leadership role with significant influence over AI strategy and team direction.

- Access to state-of-the-art GPU infrastructure and cutting-edge AI tools.

- Competitive compensation package with performance-based incentives.

- Flexible working arrangements with hybrid options.

- Continuous learning budget for conferences, courses, and certifications

📌 Senior/Lead Azure Data Engineer (Bengaluru)
🏢 Creative Synergies
📍 Bengaluru