08 Aug
|
Important Group
|
Coimbatore
08 Aug
Important Group
Coimbatore
Design and implement scalable batch and streaming data pipelines using Azure Databricks, Delta Live Tables, and Apache Spark.
Build and maintain ETL/ELT workflows orchestrated through Azure Data Factory and Databricks Workflows.
Develop reusable and modular pipeline components following software engineering best practices.
Architect and manage Lakehouse solutions using Delta Lake across Bronze, Silver, and Gold layers.
Design and enforce data models, schemas, and governance policies using Unity Catalog.
Optimize storage, partitioning, and query performance for large -scale datasets on ADLS Gen2.
Manage Databricks clusters, compute policies, and job scheduling.
Implement Infrastructure as Code (IaC) using Terraform or ARM templates.
Integrate Databricks with Azure services including Synapse, Event Hubs, Key Vault, and Azure DevOps.
Requirements
Robust proficiency in PySpark, Python,
and SQL for Big Data processing.
Hands -on experience with Delta Lake, Delta Live Tables (DLT), and Medallion Architecture.
Robust experience with Azure Data Services including:
Azure Data Lake Storage Gen2 (ADLS Gen2)
Azure Data Factory (ADF)
Azure Synapse Analytics
Azure Event Hubs
Experience with Databricks Unity Catalog for data governance and access control.
Experience implementing CI/CD pipelines using Azure DevOps or GitHub Actions for Databricks deployments.
Solid understanding of distributed computing concepts, Spark optimization, partitioning, and performance tuning.
Experience with streaming data processing using Structured Streaming, Kafka, or Event Hubs.