Remote

Document the existing flow, source interfaces, transformations, and failure modes during handover.
Automate ingestion, validation, deduplication, and incremental updates from approved sources.
Make source onboarding repeatable and reduce the validated backlog.
Implement safe retries, checkpoints, replay/backfill, and quarantine without silent loss or duplication.
Reconcile record counts and maintain metadata, source identifiers, and lineage.
Detect schema changes, apply quality checks, and investigate pipeline failures.
Maintain matching-service interfaces and contribute to agreed ML/matching changes.
Test embedding and indexing consistency, compatibility, and matching quality.
Work with DevOps on logs and alerts for freshness, failures, throughput, latency, and cost.
Maintain tests, deployment discipline, and runbooks for independent operation and recovery.
Implement approved consent, access, separation, retention, and deletion rules; report unresolved mappings.
Clarify requirements, report progress, and escalate unresolved legal decisions to the client.
Required Qualifications & Skills
5–8 years of experience owning production data pipelines, including failure investigation and recovery.
Solid SQL and PostgreSQL modelling skills,



with practical Go orchestration and migration capability.
Experience with APIs, batch and asynchronous processing, schema evolution, metadata, and lineage.
Working knowledge of idempotency, reconciliation, and secure recovery or reruns.
Production experience with vector search and embedding pipelines for search or matching,

including diagnosing input, embedding, indexing, and downstream quality issues.
Cloud operation, automated testing, code review, and deployment discipline.
Monitoring, alerts, runbooks, privacy-aware data modelling, and access controls.
Experience on data pipelines supporting production search, matching, or ML products.
Explicit communication and collaboration with Product, Engineering, and DevOps teams.
Independent investigation, practical problem-solving, and attention to detail.

Nice to have
Direct pgvector and Amazon Bedrock experience (strongly preferred).
VoyageAI embedding integrations and embedding evaluation.
Python data-processing experience.
Entity resolution and search relevance experience.
Recruitment Technology / HRTech experience and familiarity with recruitment data.

📌 Senior Data Engineer Chennai
🏢 Sourcebae
📍 Chennai