Role & responsibilities
Build and maintain data pipelines in Azure Databricks using PySpark and SQL.
• Ingest data from databases, files, APIs and streaming sources into Azure Data Lake Storage.
• Organize data in Delta Lake using Bronze, Silver and Gold layers.
• Schedule and monitor pipelines with Databricks Workflows or Azure Data Factory.
• Check data quality and fix pipeline failures quickly.
• Tune jobs and cluster settings for speed and cost.
• Set up access and permissions with Unity Catalog and Azure security tools.
• Document pipelines and work with analysts and business teams on their data needs.
Required skills:
Hands-on experience with Azure Databricks, including notebooks, clusters and jobs.
• Robust SQL and solid Python (PySpark) for transforming large data.
• Working knowledge of Delta Lake and data warehousing concepts.
• Experience with Azure services: Data Lake Storage, Data Factory and Key Vault.
• Understanding of ETL and ELT design, including batch and incremental loads.
• Version control with Git and basic CI/CD practice.
• Clear communication with both technical and non-technical colleagues.
📌 Data engineer (Gurugram)
🏢 EXL
📍 Gurugram