Role & responsibilities
Design, develop, and maintain ETL/ELT pipelines using Azure Data Factory (ADF).
Build and optimize data transformation workflows using Azure Databricks (ADB) and PySpark/Spark SQL.
Integrate data from multiple structured and unstructured sources into Azure Data Lake / Synapse.
Monitor, troubleshoot, and tune pipeline performance for reliability and cost efficiency.
Implement data quality checks, logging, and error-handling frameworks within pipelines.
Collaborate with data architects, analysts, and business stakeholders to translate requirements into technical solutions.
Ensure adherence to data security, governance, and compliance standards.