Area(s) of responsibility

We are looking for a highly skilled Databricks Data Engineer with solid hands on expertise in Databricks, PySpark, ADF, and SQL. The candidate will be responsible for architecting, developing, and optimizing up-to-date data platforms and analytical solutions on Azure. The role requires solid experience building enterprise-grade data pipelines, enabling ingestion from diverse sources, implementing complex transformations, and supporting scalable analytics initiatives.

Data Engineering Execution

• Build and maintain ingestion frameworks (ADF / Databricks / Spark)

• Implement Bronze -> Silver transformations aligned to architecture

• Architect data quality checks, schema validation, and contract rules

• Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.

• Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.

• Implement complex transformations using PySpark,



ensuring performance efficiency and code modularity.

• Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.

• Familiar with using Databricks Genie.

• Build: Dimensional models (star/snowflake schemas), Fact tables, dimensions, surrogate keys, SCD handling

• Translate Silver datasets into: Analytics-ready models, Consistent KPI definitions and business logic

• Ensure: Consistency across domains (common dimensions, conformed models), Reusability and scalability of models
Databricks & PySpark Engineering

• Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.

• Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.

• Perform performance tuning, handling bottlenecks

📌 Databricks Sr Technical Lead Data Engg Pune
🏢 Birlasoft
📍 Pune