We are looking for a highly skilled Senior Data Engineer with deep, hands-on expertise in Azure Databricks to design, build, and optimize enterprise-scale data platforms. The ideal candidate has a robust track record of building Big Data, Data Lake, Delta Lake, and Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers), and is comfortable owning the full data engineering lifecycle — from ingestion to consumption-ready data products.

Key Responsibilities:

• Design, build, and maintain scalable data pipelines on Azure Databricks using PySpark/Spark SQL for batch and streaming workloads.

• Architect and implement Lakehouse solutions using Delta Lake, ensuring ACID compliance, schema evolution, and time-travel capabilities.

• Build and own the Medallion Architecture (Bronze → Silver → Gold layers), ensuring clean, validated, and business-ready datasets at each stage.

• Develop robust ETL/ELT pipelines integrating data from multiple sources (structured, semi structured, and unstructured) into a centralized Data Lake.

• Optimize Databricks clusters, jobs,



and notebooks for performance, cost-efficiency, and scalability.

• Implement data quality, validation, and governance frameworks across all layers of the Lakehouse.

• Collaborate with data architects, analysts, and business stakeholders to translate requirements into scalable data engineering solutions.

• Work with Azure ecosystem services — Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), and Event Hubs to build end-to-end data platforms.

• Implement CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps or GitHub Actions.

• Ensure best practices around data security, access control (Unity Catalog), and compliance are followed across the platform.

• Monitor pipeline performance and troubleshoot production issues, ensuring high availability and reliability of data workflows.

• Mentor junior data engineers and contribute to engineering best practices, code r

📌 Senior Data Engineer (Navi Mumbai)
🏢 Trigent Software
📍 Navi Mumbai