Key Responsibilities
Pipeline Development: Build and maintain scalable, secure ETL/ELT pipelines using Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.
Data Processing: Utilize Apache Spark, PySpark, or Scala within Databricks to process large-scale structured and unstructured datasets.
Data Lakehouse Management: Implement Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake on Azure Data Lake Storage (ADLS) Gen2.
Data Warehousing: Design, model, and maintain high-performance data models and SQL pools in Azure Synapse.
Required Skills & Qualifications
Cloud Platforms: Expert knowledge of Microsoft Azure (ADLS Gen2, Data Factory, Synapse, Databricks).
Big Data Technologies: Proficiency in Databricks Spark, Delta Lake, and Synapse SQL.
Programming Languages: Robust proficiency in SQL, Python, or Scala.
Data Modeling: Expertise in dimensional modeling (star/snowflake schema) and data warehousing concepts.
Certifications: Preferred certification in Microsoft Certified: Azure Data Engineer Associate (DP-203)