Area(s) of responsibility

Role Summary

We are looking for a highly skilled Databricks Data Engineer with strong hands on expertise in Databricks, PySpark, ADF, and SQL. The candidate will be responsible for architecting, developing, and optimizing up-to-date data platforms and analytical solutions on Azure. The role requires strong experience building enterprise-grade data pipelines, enabling ingestion from diverse sources, implementing complex transformations, and supporting scalable analytics initiatives.

Key Responsibilities

- Data Engineering Execution

- Build and maintain ingestion frameworks (ADF / Databricks / Spark)

- Implement Bronze ->
- Silver transformations aligned to architecture

- Architect data quality checks, schema validation, and contract rules

- Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.

- Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.

- Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.

- Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.

- Familiar with using Databricks Genie.

- Build: Dimensional models (star/snowflake schemas), Fact tables, dimensions, surrogate keys, SCD handling

- Translate Silver datasets into: Analytics-ready models, Consistent KPI definitions and business logic

- Ensure: Consistency across domains (common dimensions,



conformed models), Reusability and scalability of models

- Databricks &

- PySpark Engineering

- Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.

- Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.

- Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.

- Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.

- Data Governance &

- Quality

- Implement data quality checks, audit mechanisms, and validation frameworks to ensure data accuracy and consistency in Unity Catalog.

- Enforce: Unity Catalog standards; naming conventions, metadata policies; access controls (RBAC/ABAC).

- Handle changes such as Column derivation logic changes (not just schema changes), Backward compatibility and impact analysis.

- Partner with business / data stewards to define business rules and validate metrics and edge cases.

- Collaboration &

- Stakeholder Management

- Collaborate closely with customer IT and business teams to understand data requirements and deliver reliable, production-ready solutions.

- Work with architects, product owners, and cross-functional engineering teams to align technical delivery with business objectives.

Provide guidance and mentoring to junior engineers when required

📌 Databricks & PySpark - Sr Technical Lead-Data Engg (Pune)
🏢 Birlasoft
📍 Pune