Remote

Document the existing flow, source interfaces, transformations, and failure modes during

handover.

• Automate ingestion, validation, deduplication, and incremental updates from approved sources.

• Make source onboarding repeatable and reduce the validated backlog.

• Implement protected retries, checkpoints, replay/backfill, and quarantine without silent loss or

duplication.

• Reconcile record counts and maintain metadata, source identifiers, and lineage.

• Detect schema changes, apply quality checks, and investigate pipeline failures.

• Maintain matching-service interfaces and contribute to agreed ML/matching changes.

• Test embedding and indexing consistency, compatibility, and matching quality.

• Work with DevOps on logs and alerts for freshness, failures, throughput, latency, and cost.

• Maintain tests, deployment discipline, and runbooks for independent operation and recovery.

• Implement approved consent, access, separation, retention, and deletion rules; report unresolved

mappings.

• Clarify requirements, report progress, and escalate unresolved legal decisions to the client.

4. Required Qualifications & Skills

• 5–8 years of experience owning production data pipelines, including failure investigation and

recovery.





• Strong SQL and PostgreSQL modelling skills, with practical Go orchestration and migration

capability.

• Experience with APIs, batch and asynchronous processing, schema evolution, metadata, and

lineage.

• Working knowledge of idempotency, reconciliation, and safe recovery or reruns.

• Production experience with vector search and embedding pipelines for search or matching,

including diagnosing input, embedding, indexing, and downstream quality issues.

• Cloud operation, automated testing, code review, and deployment discipline.

• Monitoring, alerts, runbooks, privacy-aware data modelling, and access controls.

• Experience on data pipelines supporting production search, matching, or ML products.

• Clear communication and collaboration with Product, Engineering, and DevOps teams.

• Independent investigation, practical problem-solving, and attention to detail.

Nice to have:

• Direct pgvector and Amazon Bedrock experience (strongly preferred).

• VoyageAI embedding integrations and embedding evaluation.

• Python data-processing experience.

• Entity resolution and search relevance experience.

• Recruitment Technology / HRTech experience and familiarity with recruitment data.

📌 Senior Data Engineer (Chennai)
🏢 Sourcebae
📍 Chennai