18 Nov
|
EliteSquad.AI
|
Mumbai
18 Nov
EliteSquad.AI
Mumbai
Azure Databricks Data Engineer
Location:
400013, Mumbai, Maharashtra, India
Experience:
5+ Years Contact / CV Submission: SANJAY.
C@ELITESQUAD.
COM
About the Role
We are looking for a skilled Data Engineer with strong hands‑on experience in Azure Data Platform, including Azure Databricks, Azure Data Factory (ADF), Azure Synapse, and modern data engineering practices. You will design, build, and optimize scalable data pipelines and collaborate closely with architects, analysts, and business teams to enable data-driven decision-making.
Key Responsibilities
• Design, develop, and maintain end‑to‑end data pipelines using Azure Databricks, ADF, and other Azure data services.
• Build scalable ETL/ELT workflows for ingesting, transforming, and loading data from structured and unstructured sources.
• Implement Delta Lake architectures, medallion designs (Bronze/Silver/Gold layers), and best practices in Databricks.
• Optimize big data processing using Spark (PySpark/Scala).
• Manage data orchestration, scheduling, and automation using ADF and Databricks Jobs.
• Work with Azure components such as Azure Data Lake Storage (ADLS), Azure SQL, Key Vault, and Azure Event Hub or Service Bus.
• Ensure data quality, reliability, lineage, and governance across the data lifecycle.
• Build CI/CD pipelines for data workloads using Azure DevOps or Git-based workflows.
• Collaborate with stakeholders to translate business needs into scalable technical solutions.
• Monitor and improve performance of data pipelines and cloud resources.
• Ensure compliance with security, privacy, and organizational data standards. Required Skills & Qualifications
• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Technology, or related field.
• 5+ years of experience as a Data Engineer or similar role.
• Solid hands‑on experience with Azure Databricks (PySpark/Scala/Spark SQL).
• Proven experience developing pipelines using Azure Data Factory.
• Expertise in Azure Data Lake, Delta Lake, and modern data lakehouse architecture.
• Solid understanding of Spark performance tuning, partitioning, caching, and optimization strategies.
• Experience with Azure DevOps, Git repos, CI/CD for data workflows.
• Proficiency in SQL, including performance tuning and complex query building.
• Knowledge of data modeling, ETL/ELT frameworks, and data warehousing concepts.
• Experience with monitoring, alerting, and cost optimization in cloud environments. Preferred / Nice-to-Have
• Experience with Azure Synapse Analytics (SQL Pools, Pipelines).
• Familiarity with Databricks Unity Catalog and advanced security controls.
• Exposure to streaming technologies (Structured Streaming, Event Hub, Kafka).
• Knowledge of Python for data engineering automation scripts.
• Understanding of MLflow, feature stores, or machine learning pipeline integration.
📌 Azure Databricks Data Engineer (Mumbai)
🏢 EliteSquad.AI
📍 Mumbai