Responsibilities
Execute Legacy-to-Cloud Migrations: Lead the end-to-end migration of complex datasets from on-premise legacy systems (including Mainframe and Informatica-based workflows) to the Azure.
Develop Databricks Pipelines: Build and optimize high-throughput ETL/ELT pipelines using PySpark and Delta Live Tables (DLT) to ensure data consistency and reliability.
Integrate Hybrid Workflows: Map and translate legacy logic (Informatica mappings and Mainframe COBOL/Copybooks) into up-to-date, code-based transformations within Azure Data Factory (ADF) and Databricks.
Performance Tuning & MLOps: Monitor and tune Databricks clusters for cost-efficiency and performance; implement CI/CD pipelines via Azure DevOps to automate data deployments.

Required Technology stack
Azure (Data Factory, ADLS Gen2, Synapse, Azure DevOps), Azure Databricks, Apache Spark (PySpark/Scala), Delta Lake,



Advanced Python, SQL, and Shell Scripting, ADF Triggers, Databricks Workflows, or Airflow

Valuable to have
Informatica PowerCenter (mappings/workflows), Mainframe (DB2, VSAM, JCL).

Experience & Qualifications
8+ years of experience in Data Engineering.
Bachelors or Masters Degree in Computer Science, Information Systems, or a related field.
Proven track record of a full-cycle migrations from on-premise settings to Azure.
Expertise in Databricks SQL and Spark core, specifically focusing on performance optimization of large-scale joins and aggregations.
Azure Certification: AZ-204 (Developer) or AZ-305 (Solutions Architect) is a significant advantage

📌 Data Engineer Azure Bengaluru
🏢 Factspan Analytics
📍 Bengaluru