Role Overview: As a Senior Consultant, you will lead the design and optimization of high-performance data processing frameworks on Azure Databricks. You will be responsible for advanced Spark tuning, implementing robust data governance through Unity Catalog, and driving the transition toward a modern Medallion architecture.
Experience:
5–7 years of experience in data engineering, with deep expertise in AzureDatabricks and Apache Spark.
Advanced experience with Delta Lake and Medallion Architecture (Bronze/Silver/Gold).
Proven track record in Spark Performance Tuning (handling data skew, shuffle tuning, etc.).
Solid experience with Unity Catalog for data governance and security.
Key Responsibilities: Technical Skills:
Design and implement scalable Medallion Architectures using Delta LiveTables (DLT) and Databricks.
Lead the optimization of complex Spark workloads to improve execution speed and reduce cloud compute costs.
Implement enterprise-grade data governance and access control using Unity Catalog.
Architect automated CI/CD pipelines for Databricks using Databricks Asset Bundles (DABs) or Terraform.
Design sophisticated data integration patterns between Databricks, Azure Synapse, and various RDBMS.
Act as a technical mentor to junior engineers, conducting code reviews and enforcing best practices.
Soft Skills:
Solid leadership and ability to drive technical delivery.
Advanced problem-solving skills for complex architectural challenges.
Excellent communication skills for interacting with stakeholders and technical leads.
Preferred Qualifications:
Bachelor’s or Master’s degree in a technical field.
Databricks Certified Data Engineer Qualified is highly desirable.
Experience with DataOps and Infrastructure as Code (Terraform).