- Strong knowledge in Azure Databricks, ADF, Synapse, Delta Lake, Unity Framework
- Coding Experience in PySpark, PySQL, Spark
- Solid understanding of SQL
- Experience in CI/CD pipeline, Git and DevOps process
- Should be well conversant with DW & ETL concepts
Design / Development Responsibilities:
- Understand business requirements and design the ETL flow accordingly
- Build ETL / ELT pipeline using Azure Databricks
- Implement Lakehouse framework
- Optimize spark jobs for performance and cost efficiency
- Ingest data from multiple sources
- Work with Azure Data Lake, Azure Data Factory for seamless integration
- Implement robust Partitioning and clustering on the schema
- Implement CI/CD pipeline
Experience range (preferred): 7-12 years Role & responsibilities