Remote

Document the existing flow, source interfaces, transformations, and failure modes during handover.

- Automate ingestion, validation, deduplication, and incremental updates from approved sources.
- Make source onboarding repeatable and reduce the validated backlog.
- Implement safe retries, checkpoints, replay/backfill, and quarantine without silent loss or duplication.
- Reconcile record counts and maintain metadata, source identifiers, and lineage.
- Detect schema changes, apply quality checks, and investigate pipeline failures.
- Maintain matching-service interfaces and contribute to agreed ML/matching changes.
- Test embedding and indexing consistency, compatibility, and matching quality.
- Work with DevOps on logs and alerts for freshness, failures, throughput, latency, and cost.
- Maintain tests, deployment discipline, and runbooks for independent operation and recovery.
- Implement approved consent, access, separation, retention, and deletion rules; report unresolved mappings.
- Clarify requirements, report progress, and escalate unresolved legal decisions to the client.
- Required Qualifications & Skills
- 5–8 years of experience owning production data pipelines, including failure investigation and recovery.
- Strong SQL and PostgreSQL modelling skills,



with practical Go orchestration and migration capability.
- Experience with APIs, batch and asynchronous processing, schema evolution, metadata, and lineage.
- Working knowledge of idempotency, reconciliation, and secure recovery or reruns.
- Production experience with vector search and embedding pipelines for search or matching,

including diagnosing input, embedding, indexing, and downstream quality issues.

- Cloud operation, automated testing, code review, and deployment discipline.
- Monitoring, alerts, runbooks, privacy-aware data modelling, and access controls.
- Experience on data pipelines supporting production search, matching, or ML products.
- Clear communication and collaboration with Product, Engineering, and DevOps teams.
- Independent investigation, practical problem-solving, and attention to detail.

Nice to have

- Direct pgvector and Amazon Bedrock experience (strongly preferred).
- VoyageAI embedding integrations and embedding evaluation.
- Python data-processing experience.
- Entity resolution and search relevance experience.
- Recruitment Technology / HRTech experience and familiarity with recruitment data.

📌 Senior Data Engineer (Chennai)
🏢 Sourcebae
📍 Chennai