Job Description - Senior Data Engineer | Azure Databricks Power BI

Position

Senior Data Engineer - Azure Databricks Power BI

Experience: 8-10 Years

Role Summary

We are looking for an experienced Senior Data Engineer with strong Azure Databricks and Power BI expertise to design, develop, migrate, and modernize enterprise data and analytics solutions.

The candidate will be responsible for end-to-end data engineering, including data ingestion, transformation, data modelling, migration, performance optimization, data quality, and integration with BI platforms.

The role requires hands-on experience in Azure Databricks, PySpark, SQL, Azure Data Lake, and Power BI, along with practical experience in legacy data-platform and BI migration initiatives.

Experience migrating Informatica workloads to Azure Databricks and IBM Cognos reports/dashboards to Power BI will be highly preferred.

Key Responsibilities

1. Data Engineering Databricks

- Design and develop scalable data pipelines using Azure Databricks, PySpark, Spark SQL, and SQL.
- Develop robust ETL/ELT pipelines for batch and incremental data processing.
- Implement Bronze, Silver, and Gold / Medallion architecture patterns.
- Build and manage Delta Lake tables and optimize Databricks workloads.
- Implement data transformation, cleansing, validation, aggregation, and enrichment processes.
- Apply Spark performance optimization techniques including partitioning, caching, broadcast joins, and optimization of data layouts.
- Develop reusable frameworks, notebooks, libraries, and data engineering components.
- Implement error handling, logging, monitoring, restartability, and operational controls.

2. Azure Data Platform

- Work with Azure services such as:
- Azure Databricks
- Azure Data Lake Storage Gen2
- Azure Data Factory
- Azure SQL Database
- Azure Key Vault
- Azure DevOps

- Design secure and scalable data ingestion and integration patterns.
- Implement appropriate authentication, authorization, secrets management, and access controls.
- Support CI/CD implementation for Databricks notebooks, code, pipelines, and deployment artifacts.

3. Data Migration Modernization

- Participate in legacy-to-cloud data platform modernization and migration programs.
- Analyze existing ETL workflows, data models, dependencies, business rules, and transformation logic.
- Convert legacy ETL processes into scalable Databricks/PySpark solutions.
- Perform source-to-target mapping and transformation-rule analysis.
- Support data reconciliation, validation, parallel runs, defect resolution, and migration sign-off.
- Develop migration utilities and reusable patterns to accelerate large-scale migration.
- Support migration planning, assessment, estimation, cutover, rollback, and hypercare activities.

4. Informatica Azure Databricks Migration - Good to Have

Hands-on experience migrating Informatica PowerCenter / Informatica ETL workloads to Azure Databricks is highly desirable.

The candidate should ideally understand:

- Informatica mappings, workflows, sessions, transformations, and dependencies.
- Conversion of Informatica mappings into PySpark / Spark SQL.
- Mapping complex transformation logic to Databricks.
- Informatica parameterization and workflow dependencies.
- Incremental load and CDC patterns.
- Source-to-target mapping and data lineage.
- ETL reconciliation and data validation.
- Performance comparison between legacy and modernized pipelines.
- Migration factory approaches for large-scale ETL modernization.

5. Power BI BI Modernization

- Develop enterprise-grade Power BI dashboards and reports.
- Build semantic models and appropriate dimensional models for reporting.
- Develop optimized DAX measures and calculated columns.




- Implement Power BI data security, including Row-Level Security (RLS).
- Optimize report, semantic model, and query performance.
- Work with business stakeholders to translate reporting requirements into effective analytical solutions.
- Support migration of legacy BI reports to Power BI.

6. Cognos Power BI Migration - Good to Have

Experience migrating IBM Cognos Analytics reports to Power BI is highly preferred.

Exposure should include:

- Cognos reports, packages, Framework Manager models, and reporting logic.
- Assessment and inventory of existing Cognos reports.
- Mapping Cognos reports to Power BI equivalents.
- Conversion of prompts, filters, calculations, drill-throughs, and reporting logic.
- Migration of Cognos semantic/reporting models to Power BI semantic models.
- DAX implementation for Cognos calculations.
- Report rationalization and identification of redundant/obsolete reports.
- Reconciliation of Cognos and Power BI outputs.
- Business-user validation and migration sign-off.

Required Technical Skills

Skill Area

Required Expertise

Azure Databricks

Strong hands-on experience

PySpark

Strong hands-on experience

Spark SQL

Strong

SQL

Advanced

Delta Lake

Strong

Azure Data Lake Gen2

Strong

Azure Data Factory

Good

Power BI

Strong hands-on experience

DAX

Strong

Data Modelling

Strong

ETL/ELT

Strong

Data Migration

Strong

Azure DevOps / Git

Good

CI/CD

Positive

Python

Good

Data Quality Reconciliation

Strong

Good-to-Have Skills

- Informatica PowerCenter
- Informatica Databricks migration
- IBM Cognos Analytics
- Cognos Power BI migration
- Legacy ETL modernization
- Legacy BI modernization
- Azure Synapse Analytics
- Microsoft Fabric
- CDC and incremental ingestion
- Unity Catalog
- Databricks Workflows
- Azure Functions / Logic Apps
- Data governance and lineage tools

Role Summary

We are looking for an experienced Senior Data Engineer with strong Azure Databricks and Power BI expertise to design, develop, migrate, and modernize enterprise data and analytics solutions.

The candidate will be responsible for end-to-end data engineering, including data ingestion, transformation, data modelling, migration, performance optimization, data quality, and integration with BI platforms.

The role requires hands-on experience in Azure Databricks, PySpark, SQL, Azure Data Lake, and Power BI, along with practical experience in legacy data-platform and BI migration initiatives.

Experience migrating Informatica workloads to Azure Databricks and IBM Cognos reports/dashboards to Power BI will be highly preferred.

Key Responsibilities

1. Data Engineering Databricks

- Design and develop scalable data pipelines using Azure Databricks, PySpark, Spark SQL, and SQL.
- Develop robust ETL/ELT pipelines for batch and incremental data processing.
- Implement Bronze, Silver, and Gold / Medallion architecture patterns.
- Build and manage Delta Lake tables and optimize Databricks workloads.
- Implement data transformation, cleansing, validation, aggregation, and enrichment processes.
- Apply Spark performance optimization techniques including partitioning, caching, broadcast joins, and optimization of data layouts.
- Develop reusable frameworks, notebooks, libraries, and data engineering components.
- Implement error handling, logging, monitoring, restartability,



and operational controls.

2. Azure Data Platform

- Work with Azure services such as:
- Azure Databricks
- Azure Data Lake Storage Gen2
- Azure Data Factory
- Azure SQL Database
- Azure Key Vault
- Azure DevOps

- Design secure and scalable data ingestion and integration patterns.
- Implement appropriate authentication, authorization, secrets management, and access controls.
- Support CI/CD implementation for Databricks notebooks, code, pipelines, and deployment artifacts.

3. Data Migration Modernization

- Participate in legacy-to-cloud data platform modernization and migration programs.
- Analyze existing ETL workflows, data models, dependencies, business rules, and transformation logic.
- Convert legacy ETL processes into scalable Databricks/PySpark solutions.
- Perform source-to-target mapping and transformation-rule analysis.
- Support data reconciliation, validation, parallel runs, defect resolution, and migration sign-off.
- Develop migration utilities and reusable patterns to accelerate large-scale migration.
- Support migration planning, assessment, estimation, cutover, rollback, and hypercare activities.

4. Informatica Azure Databricks Migration - Good to Have

Hands-on experience migrating Informatica PowerCenter / Informatica ETL workloads to Azure Databricks is highly desirable.

The candidate should ideally understand:

- Informatica mappings, workflows, sessions, transformations, and dependencies.
- Conversion of Informatica mappings into PySpark / Spark SQL.
- Mapping complex transformation logic to Databricks.
- Informatica parameterization and workflow dependencies.
- Incremental load and CDC patterns.
- Source-to-target mapping and data lineage.
- ETL reconciliation and data validation.
- Performance comparison between legacy and modernized pipelines.
- Migration factory approaches for large-scale ETL modernization.

5. Power BI BI Modernization

- Develop enterprise-grade Power BI dashboards and reports.
- Build semantic models and appropriate dimensional models for reporting.
- Develop optimized DAX measures and calculated columns.
- Implement Power BI data security, including Row-Level Security (RLS).
- Optimize report, semantic model, and query performance.
- Work with business stakeholders to translate reporting requirements into effective analytical solutions.
- Support migration of legacy BI reports to Power BI.

6. Cognos Power BI Migration - Good to Have

Experience migrating IBM Cognos Analytics reports to Power BI is highly preferred.

Exposure should include:

- Cognos reports, packages, Framework Manager models, and reporting logic.
- Assessment and inventory of existing Cognos reports.
- Mapping Cognos reports to Power BI equivalents.
- Conversion of prompts, filters, calculations, drill-throughs, and reporting logic.
- Migration of Cognos semantic/reporting models to Power BI semantic models.
- DAX implementation for Cognos calculations.
- Report rationalization and identification of redundant/obsolete reports.
- Reconciliation of Cognos and Power BI outputs.
- Business-user validation and migration sign-off.

Good-to-Have Skills

- Informatica PowerCenter
- Informatica Databricks migration
- IBM Cognos Analytics
- Cognos Power BI migration
- Legacy ETL modernization
- Legacy BI modernization
- Azure Synapse Analytics
- Microsoft Fabric
- CDC and incremental ingestion
- Unity Catalog
- Databricks Workflows
- Azure Functions / Logic Apps
- Data governance and lineage tools

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Senior Data Engineer | Azure Databricks & Power BI (Bengaluru)
🏢 Visionet Systems
📍 Bengaluru