We are looking for an experienced Azure Data Engineer to design, build, and support end-to-end data pipelines on the Azure ecosystem. The role involves extracting and transforming data from multiple source systems, building robust ADF/Synapse/Databricks pipelines, and supporting business-critical reporting and reconciliation processes.
Skills Required
Azure Data Factory (ADF)
Azure Databricks (ADB)
Azure Active Directory
Azure Storage
Azure Data Lake Services (ADLS)
Azure Key Vault
Azure Logic Apps
Azure SQL DB
Azure HDInsight
Good knowledge of Azure DevOps (ADO) services ? Repos, Boards, Build Pipelines (CI/CD)
Good knowledge of developing data engineering frameworks and notebooks using Azure Databricks with Spark SQL, PySpark, and Python
Valuable knowledge of Azure Synapse Analytics
Good-to-Have Skills
GitHub
PowerApps
Azure Monitor and Log Analytics
Power BI
REST APIs
Daily Activities
Extracting data from source systems and files in any format from SFTP locations.
Processing the extracted data and storing it into staging tables in Azure Blob Storage.
Transforming data based on the mapping (STTM) document to meet given requirements in ADB using SQL, Python, and PySpark code.
Generating a final TLM file or report and loading it into the Smartstream environment.
Building pipelines end-to-end using ADF and Synapse.
Monitoring pipelines and alerts in ADF.
Implementing recent enhancements for the LBU.
Analyzing and fixing issues raised by the business.
Updating JIRA and documenting in Confluence.
Ideal Candidate Profile
5?6 years of hands-on experience in Azure data engineering, with solid exposure to ADF, Databricks, and Synapse.
Solid programming skills in SQL, Python, and PySpark for building transformation logic and data engineering frameworks.
Experience working with SFTP-based data extraction and file-based integrations.
Familiarity with CI/CD practices using Azure DevOps.
Strong analytical and issue-resolution skills, with experience supporting business/production issues.
Good communication skills for documentation (Confluence) and task tracking (JIRA).