- We are looking for an experienced Data Engineer with 12 16 years of overall IT experience and strong expertise in Azure Databricks and PySpark
- The ideal candidate should have extensive experience in building scalable data pipelines processing large datasets and delivering enterprise grade data solutions on the Azure platform
Key Responsibilities:
- Design develop and optimize large scale data pipelines using Azure Databricks and PySpark
- Build and maintain ETL ELT workflows using Azure Data Factory
- Develop scalable data processing frameworks for batch and near real time data ingestion
- Implement data transformation cleansing and enrichment processes
- Work with large scale structured and unstructured datasets in Azure Data Lake
- Optimize Spark jobs for performance scalability and cost efficiency
- Collaborate with business stakeholders architects and analytics teams to deliver data solutions
- Ensure data quality security governance and compliance standards are met
- Provide production support troubleshoot issues and drive continuous improvements
- Mentor junior team members and contribute to technical solution design
Technical Requirements:
- 12 16 years of IT experience with strong Data Engineering expertise
- Hands on experience in Azure Databricks and PySpark development
- Strong knowledge of Azure Data Factory ADF
- Expertise in SQL and Data Warehousing concepts
- Experience with Azure Data Lake Storage ADLS Gen2
- Robust understanding of Spark architecture and performance tuning
- Experience designing and implementing enterprise scale data pipelines
- Knowledge of CI CD and Agile delivery practices
Preferred Skills:
Technology->Data Engineering->Databricks,Technology->Big Data - Data Processing->PySpark,Technology->Cloud Platform->Azure Networking Services->Azure Bastion