We are looking for a highly skilled Senior Data Engineer with deep, hands-on expertise in Azure Databricks to design, build, and optimize enterprise-scale data platforms. The ideal candidate has a strong track record of building Big Data, Data Lake, Delta Lake, and Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers), and is comfortable owning the full data engineering lifecycle - from ingestion to consumption-ready data products. Key Responsibilities:

Design, build, and maintain scalable data pipelines on Azure Databricks using PySpark/Spark SQL for batch and streaming workloads.
Architect and implement Lakehouse solutions using Delta Lake, ensuring ACID compliance, schema evolution, and time-travel capabilities.
Build and own the Medallion Architecture (Bronze Silver Gold layers), ensuring clean, validated, and business-ready datasets at each stage. Develop robust ETL/ELT pipelines integrating data from multiple sources (structured, semi structured, and unstructured) into a centralized Data Lake. Optimize Databricks clusters, jobs, and notebooks for performance, cost-efficiency, and scalability. Implement data quality, validation, and governance frameworks across all layers of the Lakehouse. Collaborate with data architects, analysts, and business stakeholders to translate requirements into scalable data engineering solutions. Work with Azure ecosystem services - Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), and Event Hubs to build end-to-end data platforms. Implement CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps or GitHub Actions. Ensure best practices around data security, access control (Unity Catalog),



and compliance are followed across the platform. Monitor pipeline performance and troubleshoot production issues, ensuring high availability and reliability of data workflows. Mentor junior data engineers and contribute to engineering best practices, code reviews, and documentation.

Required Skills & Experience:
5 7 years of overall experience in Data Engineering, with a strong focus on Big Data platforms.
Mandatory hands-on experience with Azure Databricks - this is a core requirement, not optional.
Solid experience building and managing Data Lake / Lakehouse architectures using Delta Lake.
Proven expertise implementing Medallion Architecture (Bronze/Silver/Gold) in production environments.
Strong programming skills in PySpark, Spark SQL, and Python.
Solid understanding of distributed computing concepts and Spark internals (partitioning, caching, shuffling, optimization techniques).
Hands-on experience with Azure Data Factory (ADF) for orchestration and data ingestion. Experience with Azure Data Lake Storage Gen2 (ADLS) and related Azure data services (Event Hubs, Key Vault etc.). Experience with Unity Catalog or similar data governance/security frameworks within Databricks. Strong SQL skills and experience with data modeling (dimensional modeling, star/snowflake schemas). Experience with CI/CD practices for data pipelines (Azure DevOps, Git, GitHub Actions). Familiarity with streaming data pipelines (Structured Streaming, Auto Loader, Kafka/Event Hubs) is a plus. Understanding of data quality frameworks, monitoring, and observability tools. Education Bachelor's or master's degree in computer science, Information Technology, Engineering, or a related field (or equivalent practical experience).

📌 Senior Data Engineer -Databricks || Navi Mumbai || WFO (Panvel)
🏢 TRIGENT SOFTWARE PRIVATE
📍 Panvel