17 Nov
|
Apptunix
|
Chandigarh
17 Nov
Apptunix
Chandigarh
Senior Data Engineer – Data Operations (Preference to local Chandigarh, Mohali, Punjab nearby candidates)
Experience:
5–8 Years
Location:
Remote Engagement: 6-Month Contract (Extendable)
Working Hours
8 Hours/Day with UK Time-Zone Overlap Role Overview We are looking for a Senior Data Engineer with robust hands-on experience in building, managing, and troubleshooting production-grade data systems. The role requires advanced expertise in pipeline reliability and recovery, PostgreSQL optimisation, data orchestration, and AI-powered embedding and vector-search implementations.
Key Responsibilities
• Design, develop, and maintain production ETL/ELT and data ingestion pipelines, including batch and incremental processing.
• Implement robust pipeline recovery mechanisms, including retries, checkpoints, replay, backfills, reconciliation, idempotency, and failure handling.
• Optimise PostgreSQL queries, indexing, database schemas, and data models for performance and scalability.
• Build and manage data orchestration workflows using DAG-based tools such as Apache Airflow, Dagster, or equivalent.
• Implement and troubleshoot embedding generation, vector indexing, semantic search, and retrieval pipelines.
• Manage schema evolution, data validation, deduplication, metadata, and data lineage.
• Integrate data through REST APIs and asynchronous processing workflows.
• Monitor production systems through logging, alerts, testing, deployment, and operational runbooks.
• Ensure secure data access, privacy, and reliability across data systems.
Required Technical Skills
• Data Engineering: Production ETL/ELT, ingestion pipelines, batch processing, incremental loads, data transformation, validation, deduplication.
• Pipeline Reliability: Retries, checkpoints, replay, backfills, idempotency, reconciliation, recovery, and failure management.
• Databases: Advanced SQL, PostgreSQL optimisation, query performance tuning, indexing, relational data modelling.
• Orchestration: Apache Airflow, Dagster, or equivalent DAG-based workflow orchestration.
• Programming: Practical Go (Golang) for orchestration, automation, and data migration; Python is an added advantage.
• AI Search: Hands-on embedding generation, vector databases, vector indexing, semantic search, retrieval pipelines, and troubleshooting search relevance.
• Data Management: Schema evolution, metadata management, lineage, APIs, and asynchronous processing.
• Infrastructure: Cloud platforms, Docker, CI/CD, monitoring, testing, and production deployment. Preferred Experience
• Experience with pgvector, Amazon Bedrock, and VoyageAI.
• Experience with AI-powered search, entity resolution, and search relevance optimisation.
• Exposure to recruitment technology, HRTech, or large-scale data-driven applications. Key Selection Focus: Candidates must demonstrate real-world ownership of production data pipelines, complex recovery scenarios, PostgreSQL performance optimisation, and hands-on embedding/vector-search implementations—not just theoretical or project-level exposure.
📌 Data Engineer (Chandigarh)
🏢 Apptunix
📍 Chandigarh