17 Nov
|
Apptunix
|
India
Senior Data Engineer – Data Operations (Preference to local Chandigarh, Mohali, Punjab nearby candidates)
Experience: 5–8 Years
Location: Remote
Engagement: 6-Month Contract (Extendable)
Working Hours: 8 Hours/Day with UK Time-Zone Overlap
Role Overview
We are looking for a Senior Data Engineer with solid hands-on experience in building, managing, and troubleshooting production-grade data systems. The role requires advanced expertise in pipeline reliability and recovery, PostgreSQL optimisation, data orchestration, and AI-powered embedding and vector-search implementations.
Key Responsibilities
- Design, develop, and maintain production ETL/ELT and data ingestion pipelines, including batch and incremental processing.
- Implement robust pipeline recovery mechanisms, including retries, checkpoints, replay, backfills, reconciliation, idempotency, and failure handling.
- Optimise PostgreSQL queries, indexing, database schemas, and data models for performance and scalability.
- Build and manage data orchestration workflows using DAG-based tools such as Apache Airflow, Dagster, or equivalent.
- Implement and troubleshoot embedding generation, vector indexing, semantic search, and retrieval pipelines.
- Manage schema evolution, data validation, deduplication, metadata, and data lineage.
- Integrate data through REST APIs and asynchronous processing workflows.
- Monitor production systems through logging, alerts, testing, deployment, and operational runbooks.
- Ensure secure data access, privacy, and reliability across data systems.
Required Technical Skills
- Data Engineering: Production ETL/ELT, ingestion pipelines, batch processing, incremental loads, data transformation, validation, deduplication.
- Pipeline Reliability: Retries, checkpoints, replay, backfills, idempotency, reconciliation, recovery, and failure management.
- Databases: Advanced SQL, PostgreSQL optimisation, query performance tuning, indexing, relational data modelling.
- Orchestration: Apache Airflow, Dagster, or equivalent DAG-based workflow orchestration.
- Programming: Practical Go (Golang) for orchestration, automation, and data migration; Python is an added advantage.
- AI Search: Hands-on embedding generation, vector databases, vector indexing, semantic search, retrieval pipelines, and troubleshooting search relevance.
- Data Management: Schema evolution, metadata management, lineage, APIs, and asynchronous processing.
- Infrastructure: Cloud platforms, Docker, CI/CD, monitoring, testing, and production deployment.
Preferred Experience
- Experience with pgvector, Amazon Bedrock, and VoyageAI.
- Experience with AI-powered search, entity resolution, and search relevance optimisation.
- Exposure to recruitment technology, HRTech, or large-scale data-driven applications.
Key Selection Focus: Candidates must demonstrate real-world ownership of production data pipelines, complex recovery scenarios, PostgreSQL performance optimisation, and hands-on embedding/vector-search implementations—not just theoretical or project-level exposure.
📌 Data Engineer (India)
🏢 Apptunix
📍 India