Key Responsibilities
- Lead the design, architecture, and implementation of Azure Databricks solutions for data processing and analytics.
- Develop and optimize ETL/ELT pipelines using Apache Spark, Python (PySpark), Scala, and SQL.
- Work closely with data engineers, data scientists, and business stakeholders to ensure efficient data workflows.
- Implement data governance, security, and compliance best practices in Databricks and Azure Data Lake.
- Optimize performance tuning, cost management, and scalability of Databricks workloads.
- Automate CI/CD pipelines for Databricks notebooks using DevOps tools like Azure DevOps, Terraform, or GitHub Actions.
- Troubleshoot and resolve issues related to big data processing, Spark jobs, and cloud storage.
- Stay updated with Azure and Databricks best practices and mentor junior team members.
- Required Skills & Experience:

-

-

10+ years of experience in data engineering, big data, or cloud-based data solutions.
- Solid hands-on experience with Azure Databricks, Apache Spark, and Delta Lake.
- Expertise in Python (PySpark), Scala, SQL,



and data transformation techniques.
- Deep understanding of Azure Data Services (Azure Data Lake, Azure Synapse, Azure Data Factory, etc.).
- Experience with data modeling, ETL/ELT design, and distributed data processing.
- Proficiency in DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and version control (Git).
- Familiarity with MLflow, Unity Catalog, and Databricks SQL is a plus.
- Strong analytical, problem-solving, and communication skills.

-

-

Preferred Qualifications
- Databricks certifications (e.g., Databricks Certified Data Engineer Associate/Professional).
- Experience with containerization (Docker, Kubernetes) and orchestration tools.
- Knowledge of real-time data streaming using Kafka, Event Hubs, or similar technologies.
- Exposure to Machine Learning (ML) workflows in Databricks is a plus.
- Competitive Salary
- Performance based Variable Pay
-

📌 Data and Analytics (India)
🏢 Orion India Systems Private
📍 India