Job Summary: We are looking for a highly skilled Senior Azure Databricks Engineer to lead the design, development, and optimization of our enterprise Data Lakehouse. This role bridges the gap between traditional data engineering and up-to-date cloud scale, focusing on building high-performance, reliable, and secure data pipelines. You will be instrumental in implementing the Medallion Architecture and ensuring that our data ecosystem supports both real-time streaming and complex batch analytics for downstream consumers.

Key Responsibilities: Lakehouse Development: Design and maintain scalable ETL/ELT pipelines using PySpark and Spark SQL in both batch and streaming modes.

Medallion Architecture: Implement and govern the Bronze, Silver, and Gold layers using Delta Lake, ensuring ACID compliance, schema enforcement, and data versioning (Time Travel).

Performance Engineering: Optimize Spark jobs by managing partitions, caching strategies, and broadcast joins.

Leverage Adaptive Query

Execution (AQE) and Z-Ordering to minimize compute latency and cost.

Stream Processing: Build and manage seamless data flows using Apache Kafka or Confluent to support real-time data ingestion and event-driven architectures. Governance &

- Security:



Configure and manage Unity Catalog for fine-grained access control, automated data lineage, and comprehensive audit logging.

Orchestration: Automate complex workflows using Databricks Workflows, Delta Live Tables (DLT), or Azure Data Factory (ADF). Collaboration &
- Mentorship: Partner with Data Architects and Scientists to translate business logic into technical specifications. Provide technical mentorship to junior engineers and drive engineering best practices.

Required Technical Skills: Databricks Ecosystem: Mastery of the Databricks Lakehouse Platform, including Delta Lake, Unity Catalog, and Photon engine.

Programming: Advanced proficiency in Python (PySpark) and SQL for complex data transformations; exposure to Scala is a plus. Streaming &

- Integration: Proven experience with Kafka and integrating Databricks with Azure services like ADLS Gen2, Cosmos DB, and Azure Key Vault. Database &
- Storage: Deep knowledge of NoSQL databases (Cosmos DB, MongoDB) and file formats like Parquet and Avro.DevOps &
- CI/CD: Familiarity with Git-based workflows, Terraform for Infrastructure as Code (IaC), and automated deployment pipelines.

📌 Lead Software Engineer (Chennai)
🏢 Virtusa
📍 Chennai