- Azure Services
- Azure Data Factory ADF
- Azure Databricks
- Azure Data Lake Storage ADLS Gen2
- Azure SQL Database SQL Server
- Azure Synapse Analytics preferred
- Programming Querying
- PySpark
- Spark SQL
- SQL Advanced
- Python
- Data Engineering
- ETL ELT Development
- Data Modeling
- Performance Tuning
- Data Warehousing Concepts
- DevOps Version Control
- Azure DevOps
- Git GitHub
Key Responsibilities:
- Design and develop ETL ELT pipelines using Azure Data Factory ADF
- Build and optimize data processing workflows using Azure Databricks PySpark Spark SQL
- Integrate data from various sources such as SQL Server Azure SQL ADLS APIs and third party applications
- Implement data transformation cleansing and validation processes
- Develop and maintain data lakes and data warehouse solutions on Azure
- Monitor and troubleshoot pipeline failures performance bottlenecks and data quality issues
- Collaborate with cross functional teams to understand business requirements and translate them into technical solutions
- Implement security governance and best practices for data management
- Create technical documentation and support deployment activities
- Required Skills
- Azure Services
- Azure Data Factory ADF
- Azure Databricks
- Azure Data Lake Storage ADLS Gen2
- Azure SQL Database SQL Server
- Azure Synapse Analytics preferred
Preferred Skills:
Technology->Cloud Integration->Azure Data Factory (ADF),Technology->Data Engineering->Databricks