Role Overview: As a Senior Consultant, you will lead the design and optimization of high-performance data processing frameworks on Azure Databricks. You will be responsible for advanced Spark tuning, implementing robust data governance through Unity Catalog, and driving the transition toward a modern Medallion architecture.
Experience:
- 5–7 years of experience in data engineering, with deep expertise in AzureDatabricks and Apache Spark.
- Advanced experience with Delta Lake and Medallion Architecture (Bronze/Silver/Gold).
- Proven track record in Spark Performance Tuning (handling data skew, shuffle tuning, etc.).
- Strong experience with Unity Catalog for data governance and security.
Key Responsibilities: Technical Skills:
- Design and implement scalable Medallion Architectures using Delta LiveTables (DLT) and Databricks.
- Lead the optimization of complex Spark workloads to improve execution speed and reduce cloud compute costs.
- Implement enterprise-grade data governance and access control using Unity Catalog.
- Architect automated CI/CD pipelines for Databricks using Databricks Asset Bundles (DABs) or Terraform.
- Design sophisticated data integration patterns between Databricks, Azure Synapse, and various RDBMS.
- Act as a technical mentor to junior engineers, conducting code reviews and enforcing best practices.
Soft Skills:
- Strong leadership and ability to drive technical delivery.
- Advanced problem-solving skills for complex architectural challenges.
- Excellent communication skills for interacting with stakeholders and technical leads.
Preferred Qualifications:
- Bachelor’s or Master’s degree in a technical field.
- Databricks Certified Data Engineer Qualified is highly desirable.
- Experience with DataOps and Infrastructure as Code (Terraform).