Job highlights

- Expertise in Databricks platform, Unity Catalog, Delta Lake, CI/CD frameworks, and cloud architecture (AWS/Azure)
- Define and lead enterprise Databricks strategy, implement governance, design data lakehouse and pipelines, manage portfolio and lead data engineering teams

Key Responsibilities 1. Databricks Enterprise Strategy

- Define and own the enterprise-wide Databricks platform strategy, roadmap, and adoption model in line with business goals.
- Set up a Databricks Center of Excellence (CoE), including best practices, reusable frameworks, and engineering standards.
- Assess new Databricks capabilities (Unity Catalog, Delta Live Tables, Serverless, Photon, Mosaic AI, Databricks SQL) and lead their adoption.
- Drive platform cost optimization (FinOps), workspace strategy, and capacity planning.
- Act as the primary technical liaison with Databricks and cloud vendor partners.

2. Advanced DABs (Databricks Asset Bundles) Framework

- Design and implement an enterprise-grade CI/CD framework using Databricks Asset Bundles for consistent, repeatable deployments.
- Standardize Infrastructure-as-Code and Configuration-as-Code practices across environments (Dev, QA, UAT, Prod).
- Integrate DABs with DevOps tools such as Azure DevOps, GitHub Actions, GitLab CI, Jenkins, and Terraform.
- Build reusable bundle templates, deployment patterns, and automated testing and validation pipelines.
- Ensure version control, setting promotion, and release governance across all Databricks workloads.

3. Governance Implementation

- Lead the design and rollout of data governance using Unity Catalog, covering access controls, data lineage, auditing, and data discovery.
- Implement data security, privacy,



and compliance controls (e.g., GDPR, HIPAA, SOC 2, CCPA, as applicable).
- Define data quality frameworks, metadata management standards, and data ownership and stewardship models.
- Work with InfoSec, Risk, and Compliance teams to enforce policies such as row/column-level security, data masking, and fine-grained permissions.
- Set up monitoring, observability, and audit mechanisms across the data platform.

4. Data Lakehouse Architecture

- Architect and evolve the enterprise Data Lakehouse using Delta Lake and the Medallion Architecture (Bronze, Silver, Gold).
- Design scalable data models that support analytics, BI, ML/AI, and real-time use cases.
- Lead migrations from legacy data warehouses, Hadoop, and on-prem platforms to the Databricks Lakehouse.
- Ensure performance tuning, storage optimization (Z-Ordering, Liquid Clustering, partitioning), and high availability.
- Enable data sharing through Delta Sharing and support data mesh and data product approaches where relevant.

5. Cloud Architecture (AWS / Azure)

- Design secure, scalable, and resilient cloud-native data architectures on AWS and/or Azure.
- Oversee the integration of Databricks with cloud services, such as:
- Azure: ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Synapse, Entra ID, Purview
- AWS: S3, Glue,



Kinesis, Lambda, IAM, Redshift, Lake Formation
- Define networking, identity, and security architecture, including VNet/VPC injection, Private Link, and SSO/SCIM.
- Drive disaster recovery, business continuity, and multi-region strategies.

6. Enterprise Data Pipeline Design

- Lead the design and delivery of robust batch and streaming pipelines using PySpark, Spark SQL, Structured Streaming, Delta Live Tables, and Auto Loader.
- Establish reusable ingestion, transformation, and orchestration frameworks (Databricks Workflows, Airflow, ADF).
- Ensure pipelines meet SLAs, scalability, reliability, and data quality requirements.
- Promote engineering excellence through code reviews, testing standards, modular design, and documentation.
- Enable real-time and near-real-time data processing for critical business use cases.

7. Portfolio Management

- Manage a portfolio of data engineering programs and projects, ensuring delivery on scope, schedule, budget, and quality.
- Prioritize initiatives based on business value, ROI, and strategic alignment.
- Own resource planning, budgeting, vendor management, and risk mitigation across the portfolio.
- Report progress, KPIs, and outcomes to senior leadership and executive stakeholders.
- Run delivery governance using Agile, SAFe, or hybrid methodologies.

People Leadership

- Lead, mentor, and grow a team of data engineers, architects, and tech leads, including onshore and offshore teams.
- Build a culture of innovation, accountability, and continuous learning.
- Drive hiring, performance management, and career development for the team.

📌 Data Engineer (Databricks ) Senior Manager (Pune)
🏢 PwC
📍 Pune