Key Responsibilities
Lakehouse Architecture: Design end-to-end Medallion architecture (Bronze, Silver, Gold layers) using Databricks Delta Lake.
Pipeline Governance: Lead data governance, security, and cataloging strategies across the enterprise using Unity Catalog.
Performance Tuning: Optimize Spark clusters, Delta Live Tables (DLT), and Databricks SQL queries for maximum cost-efficiency.
Data Engineering Leadership: Establish CI/CD workflows, guide development teams on PySpark best practices, and translate business requirements into technical blueprints.
Required Qualifications & Skills
Core Tech: Deep expertise in Databricks, PySpark, Spark SQL, Delta Lake, and Unity Catalog.
Cloud Platform: Hands-on architecture experience in (Azure / AWS / GCP).
DevOps & Integration: Proficiency with Git, CI/CD tools, orchestration (Airflow/ADF), and BI systems (Power BI/Tableau).
Certifications: Databricks Certified Data Architect or Solutions Architect is highly preferred.