Role

We are looking for a highly skilled Senior Data Engineer with deep, hands-on expertise in Azure Databricks to design, build, and optimize enterprise-scale data platforms. The ideal candidate has a strong track record of building Big Data, Data Lake, Delta Lake, and Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers), and is comfortable owning the full data engineering lifecycle — from ingestion to consumption-ready data products.

Key Responsibilities

- Design, build, and maintain scalable data pipelines on Azure Databricks using PySpark/Spark SQL for batch and streaming workloads.
- Architect and implement Lakehouse solutions using Delta Lake, ensuring ACID compliance, schema evolution, and time-travel capabilities.
- Build and own the Medallion Architecture (Bronze → Silver → Gold layers), ensuring clean, validated, and business-ready datasets at each stage.
- Develop robust ETL/ELT pipelines integrating data from multiple sources (structured, semi structured, and unstructured) into a centralized Data Lake.
- Optimize Databricks clusters, jobs, and notebooks for performance, cost-efficiency, and scalability.
- Implement data quality, validation, and governance frameworks across all layers of the Lakehouse.
- Collaborate with data architects, analysts, and business stakeholders to translate requirements into scalable data engineering solutions.
- Work with Azure ecosystem services — Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), and Event Hubs to build end-to-end data platforms.
- Implement CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps or GitHub Actions.
- Ensure best practices around data security, access control (Unity Catalog),



and compliance are followed across the platform.
- Monitor pipeline performance and troubleshoot production issues, ensuring high availability and reliability of data workflows.
- Mentor junior data engineers and contribute to engineering best practices, code reviews, and documentation.

Required Skills &

Experience

- 5–7 years of overall experience in Data Engineering, with a strong focus on Big Data platforms.
- Mandatory hands-on experience with Azure Databricks - this is a core requirement, not optional.
- Robust experience building and managing Data Lake / Lakehouse architectures using Delta Lake.
- Proven expertise implementing Medallion Architecture (Bronze/Silver/Gold) in production environments.
- Strong programming skills in PySpark, Spark SQL, and Python.
- Solid understanding of distributed computing concepts and Spark internals (partitioning, caching, shuffling, optimization techniques).
- Hands-on experience with Azure Data Factory (ADF) for orchestration and data ingestion.
- Experience with Azure Data Lake Storage Gen2 (ADLS) and related Azure data services (Event Hubs, Key Vault etc.).
- Experience with Unity Catalog or similar data governance/security frameworks within Databricks.
- Strong SQL skills and experience with data modeling (dimensional modeling, star/snowflake schemas).
- Experience with CI/CD practices for data pipelines (Azure DevOps, Git, GitHub Actions).
- Familiarity with streaming data pipelines (Structured Streaming, Auto Loader, Kafka/Event Hubs) is a plus.
- Understanding of data quality frameworks, monitoring, and observability tools.

Education

- Bachelor's or master's degree in computer science, Information Technology, Engineering, or a related field (or equivalent practical experience).

📌 Senior Data Engineer -Databricks || Navi Mumbai || WFO
🏢 Trigent Software
📍 Navi Mumbai