Responsiblities

Build batch-seed and event-tail ingestion per source system, including seed tail watermark handoff, idempotent upserts, and dedup ledgers

Build and operate medallion layers with reprocess-from-Bronze, pipeline orchestration (checkpoints, retry/backoff, DLQ), and full observability

Build data-quality gates (quarantine / pass-with-flag), quality scoring, and a reconciliation engine covering count, record, and financial reconciliation financial is zero-tolerance

Build identity matching combining deterministic rules with probabilistic scoring and confidence bands; deliver deduplication, golden-record materialization, and survivorship rules, calibrating match thresholds with labelled data

Author and maintain source canonical structural mappings and value crosswalks (e.g., collapsing 1,800+ raw employment-status values to ~20 standard ones) as governed, versioned configuration

Enforce data contracts at the boundary: schema registry, fail-quick validation, and semver-compatible schema evolution What We re Looking For 5+ years building production data pipelines at scale

Kafka depth: consumers/producers, replay, DLQ, exactly-once / idempotent processing patterns Strong SQL and solid ETL fundamentals





Java and/or Python in production

Medallion/lakehouse layering, CDC, watermark/checkpoint patterns, and batch stream hand-off

Data-quality frameworks: validation rules, quarantine and re-entry, quality scoring, reconciliation

Entity resolution / MDM exposure: record matching, dedup, survivorship via commercial tools (Informatica MDM, Reltio) or custom builds

Data mapping and crosswalk discipline: profiling messy datasets, authoring governed reference data, config-as-code (YAML/JSON, Git) Bonus Points

Probabilistic record linkage at depth blocking/candidate generation, scoring models, threshold calibration (expected at senior level)

Schema registry experience (Avro/Protobuf)

Extracting from mainframe or older RDBMS sources with limited CDC support

Financial reconciliation in finance-adjacent domains

Benefits administration or healthcare domain knowledge

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Data Pipeline Engineer-Kafka(SJ-BY) (Kochi)
🏢 SE Mentor
📍 Kochi