We are looking for a skilled PySpark Developer with experience in Azure Databricks (ADB) and Azure Data Factory (ADF) to join our team. The ideal candidate will play a crucial role in designing, developing, and implementing data solutions using PySpark for large-scale data processing and analytics.
Responsibilities :
- Design, develop, and deploy PySpark applications and workflows on Azure Databricks for data transformation, cleansing, and aggregation.
- Implement data pipelines using Azure Data Factory (ADF) to orchestrate ETL/ELT processes across heterogeneous data sources.
- Collaborate with Data Engineers and Data Scientists to integrate and process structured and unstructured data sets into actionable insights.
- Optimize PySpark jobs and data pipelines for performance, scalability, and reliability.
- Conduct regular financial risk assessments to identify potential vulnerabilities in data processing workflows.
- Ensure data quality and integrity throughout all stages of data processing.
- Develop and implement strategies to mitigate financial risks associated with data transformation and aggregation.
- Troubleshoot and debug issues related to data pipelines and processing.
- Ensure compliance with regulatory requirements and industry standards in all data processing activities.
- Implement best practices for data security, compliance, and privacy within Azure environment.
- Document technical specifications, data flows, and solution architecture.
Requirements :
- Bachelors degree in computer science, Engineering, or a related field; Masters degree preferred.
- Proven experience as a PySpark Developer or similar role with a strong understanding of Apache Spark internals.
- Hands-on experience with Azure Databricks (ADB) and Azure Data Factory (ADF).
- Proficiency in Python programming language and solid understanding of SQL.
- Experience designing and optimizing data pipelines for ETL/ELT processes.
- Familiarity with cloud platforms, preferably Microsoft Azure.
- Excellent problem-solving skills and ability to think critically.
- Robust communication skills with the ability to collaborate effectively in a team environment.
- Experience in Financial, Risk, Compliance, or Banking domains is a plus.
- Experience identifying and mitigating financial risks in data processes.
- Ability to analyse data for potential risk factors and develop strategies to minimize financial risk.
- Ensure all data processes comply with relevant regulatory requirements and industry standards.
Preferred Qualifications :
- Certification in Azure Data Engineering or related field.
- Knowledge of other big data technologies such as Hadoop, Hive, or Kafka.
- Familiarity with machine learning frameworks and techniques