Responsibilities

- Execute Legacy-to-Cloud Migrations: Lead the end-to-end migration of complex datasets from on-premise legacy systems (including Mainframe and Informatica-based workflows) to the Azure.
- Develop Databricks Pipelines: Build and optimize high-throughput ETL/ELT pipelines using PySpark and Delta Live Tables (DLT) to ensure data consistency and reliability.
- Integrate Hybrid Workflows: Map and translate legacy logic (Informatica mappings and Mainframe COBOL/Copybooks) into up-to-date, code-based transformations within Azure Data Factory (ADF) and Databricks.
- Performance Tuning & MLOps: Monitor and tune Databricks clusters for cost-efficiency and performance; implement CI/CD pipelines via Azure DevOps to automate data deployments.

Required Technology stack

- Azure (Data Factory, ADLS Gen2, Synapse, Azure DevOps), Azure Databricks, Apache Spark (PySpark/Scala), Delta Lake,



Advanced Python, SQL, and Shell Scripting, ADF Triggers, Databricks Workflows, or Airflow

Good to have

- Informatica PowerCenter (mappings/workflows), Mainframe (DB2, VSAM, JCL).

Experience & Qualifications

- 8+ years of experience in Data Engineering.
- Bachelors or Masters Degree in Computer Science, Information Systems, or a related field.
- Proven track record of a full-cycle migrations from on-premise environments to Azure.
- Expertise in Databricks SQL and Spark core, specifically focusing on performance optimization of large-scale joins and aggregations.
- Azure Certification: AZ-204 (Developer) or AZ-305 (Solutions Architect) is a significant advantage

📌 Data Engineer_Azure (Bengaluru)
🏢 Factspan Analytics
📍 Bengaluru