21 Nov
|
Apptunix
|
Chandigarh
21 Nov
Apptunix
Chandigarh
Senior Data Engineer – Data Operations (Preference to local Chandigarh, Mohali, Punjab nearby candidates) Experience: 5–8 Years
Location: Remote
Engagement: 6-Month Contract (Extendable)
Working Hours: 8 Hours/Day with UK Time-Zone Overlap
Role Overview
We are looking for a Senior Data Engineer with solid hands-on experience in building, managing, and troubleshooting production-grade data systems. The role requires advanced expertise in pipeline reliability and recovery, PostgreSQL optimisation, data orchestration, and AI-powered embedding and vector-search implementations.
Key Responsibilities
Design, develop, and maintain production ETL/ELT and data ingestion pipelines, including batch and incremental processing.
Implement robust pipeline recovery mechanisms, including retries, checkpoints, replay, backfills, reconciliation, idempotency, and failure handling.
Optimise PostgreSQL queries, indexing, database schemas, and data models for performance and scalability.
Build and manage data orchestration workflows using DAG-based tools such as Apache Airflow, Dagster, or equivalent.
Implement and troubleshoot embedding generation, vector indexing, semantic search, and retrieval pipelines.
Manage schema evolution, data validation, deduplication, metadata, and data lineage.
Integrate data through REST APIs and asynchronous processing workflows.
Monitor production systems through logging, alerts, testing, deployment, and operational runbooks.
Ensure secure data access, privacy, and reliability across data systems.
Required Technical Skills
Data Engineering: Production ETL/ELT, ingestion pipelines, batch processing, incremental loads, data transformation, validation, deduplication.
Pipeline Reliability: Retries, checkpoints, replay, backfills, idempotency, reconciliation, recovery, and failure management.
Databases: Advanced SQL, PostgreSQL optimisation, query performance tuning, indexing, relational data modelling.
Orchestration: Apache Airflow, Dagster, or equivalent DAG-based workflow orchestration.
Programming: Practical Go (Golang) for orchestration, automation, and data migration; Python is an added advantage.
AI Search: Hands-on embedding generation, vector databases, vector indexing, semantic search, retrieval pipelines, and troubleshooting search relevance.
Data Management: Schema evolution, metadata management, lineage, APIs, and asynchronous processing.
Infrastructure: Cloud platforms, Docker, CI/CD, monitoring, testing, and production deployment.
Preferred Experience
Experience with pgvector, Amazon Bedrock, and VoyageAI.
Experience with AI-powered search, entity resolution, and search relevance optimisation.
Exposure to recruitment technology, HRTech, or large-scale data-driven applications.
Key Selection Focus: Candidates must demonstrate real-world ownership of production data pipelines, complex recovery scenarios, PostgreSQL performance optimisation, and hands-on embedding/vector-search implementations—not just theoretical or project-level exposure.
📌 Data Engineer SQL - Remote (m/w/d) (Chandigarh)
🏢 Apptunix
📍 Chandigarh