19 Nov
|
Trigent Software
|
Navi Mumbai
19 Nov
Trigent Software
Navi Mumbai
Role:
We are looking for a highly skilled Senior Data Engineer with deep, hands-on expertise in Azure Databricks to design, build, and optimize enterprise-scale data platforms. The ideal candidate has a solid track record of building Big Data, Data Lake, Delta Lake, and Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers), and is comfortable owning the full data engineering lifecycle — from ingestion to consumption-ready data products.
Key Responsibilities
• Design, build, and maintain scalable data pipelines on Azure Databricks using PySpark/Spark SQL for batch and streaming workloads.
• Architect and implement Lakehouse solutions using Delta Lake, ensuring ACID compliance, schema evolution, and time-travel capabilities.
• Build and own the Medallion Architecture (Bronze → Silver → Gold layers), ensuring clean, validated, and business-ready datasets at each stage.
• Develop robust ETL/ELT pipelines integrating data from multiple sources (structured, semi structured, and unstructured) into a centralized Data Lake.
• Optimize Databricks clusters, jobs, and notebooks for performance, cost-efficiency, and scalability.
• Implement data quality, validation, and governance frameworks across all layers of the Lakehouse.
• Collaborate with data architects, analysts, and business stakeholders to translate requirements into scalable data engineering solutions.
• Work with Azure ecosystem services — Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), and Event Hubs to build end-to-end data platforms.
• Implement CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps or GitHub Actions.
• Ensure best practices around data security, access control (Unity Catalog),
and compliance are followed across the platform.
• Monitor pipeline performance and troubleshoot production issues, ensuring high availability and reliability of data workflows.
• Mentor junior data engineers and contribute to engineering best practices, code reviews, and documentation.
Required Skills &
Experience:
• 5–7 years of overall experience in Data Engineering, with a strong focus on Big Data platforms.
• Mandatory hands-on experience with Azure Databricks
- this is a core requirement, not optional.
• Strong experience building and managing Data Lake / Lakehouse architectures using Delta Lake.
• Proven expertise implementing Medallion Architecture (Bronze/Silver/Gold) in production environments.
• Strong programming skills in PySpark, Spark SQL, and Python.
• Solid understanding of distributed computing concepts and Spark internals (partitioning, caching, shuffling, optimization techniques).
• Hands-on experience with Azure Data Factory (ADF) for orchestration and data ingestion.
• Experience with Azure Data Lake Storage Gen2 (ADLS) and related Azure data services (Event Hubs, Key Vault etc.).
• Experience with Unity Catalog or similar data governance/security frameworks within Databricks.
• Strong SQL skills and experience with data modeling (dimensional modeling, star/snowflake schemas).
• Experience with CI/CD practices for data pipelines (Azure DevOps, Git, GitHub Actions).
• Familiarity with streaming data pipelines (Structured Streaming, Auto Loader, Kafka/Event Hubs) is a plus.
• Understanding of data quality frameworks, monitoring, and observability tools. Education
• Bachelor's or master's degree in computer science, Information Technology, Engineering, or a related field (or equivalent practical experience).
📌 Senior Data Engineer (Navi Mumbai)
🏢 Trigent Software
📍 Navi Mumbai