Azure data bricks with fact & dimension table, Spark SQL, Pyspark, autoloader, unity catalogue & file format JSon - Mandatory

Mandate Skills : Hands on experience in Azure Databricks & Medallion Architecture

7 years - 9 years. in

Azure Databricks technologies.

Experience in

Medallion Architecture

is mandatory.• Valuable understanding of ETL , BI and DW technologies.• Hands-on experience on

Jason or XML files

as sources.• Experience in

autoloader for incremental loading and explode options for flattening data• Experience in loading

Delta live tables• Experience in

PySpark

or

Spark

using data frames.• Design, implement, and maintain data pipelines for data ingestion, processing,



and transformation in Azure.• Work together with data scientists and analysts to understand the needs for data and create effective data workflows.• Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.• Utilizing Azure Data Bricks to create and maintain

ETL

(Extract, Transform, Load) operations.• Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.• Improve the scalability, efficiency, and cost-effectiveness of data pipelines.• Monitoring and resolving data pipeline problems to guarantee consistency and availability of the data.• Good communication skills – Oral & Written

📌 Data Engineer (Mumbai)
🏢 Tech Mahindra
📍 Mumbai