Key Responsibilities

- Lead the design, architecture, and implementation of Azure Databricks solutions for data processing and analytics.
- Develop and optimize ETL/ELT pipelines using Apache Spark, Python (PySpark), Scala, and SQL.
- Work closely with data engineers, data scientists, and business stakeholders to ensure efficient data workflows.
- Implement data governance, security, and compliance best practices in Databricks and Azure Data Lake.
- Optimize performance tuning, cost management, and scalability of Databricks workloads.
- Automate CI/CD pipelines for Databricks notebooks using DevOps tools like Azure DevOps, Terraform, or GitHub Actions.
- Troubleshoot and resolve issues related to big data processing, Spark jobs, and cloud storage.
- Stay updated with Azure and Databricks best practices and mentor junior team members.
- Required Skills & Experience:
- - 10+ years of experience in data engineering, big data, or cloud-based data solutions.
- Strong hands-on experience with Azure Databricks, Apache Spark, and Delta Lake.
- Expertise in Python (PySpark), Scala, SQL, and data transformation techniques.
- Deep understanding of Azure Data Services (Azure Data Lake,



Azure Synapse, Azure Data Factory, etc.).
- Experience with data modeling, ETL/ELT design, and distributed data processing.
- Proficiency in DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and version control (Git).
- Familiarity with MLflow, Unity Catalog, and Databricks SQL is a plus.
- Strong analytical, problem-solving, and communication skills.
- - Preferred Qualifications
- Databricks certifications (e.g., Databricks Certified Data Engineer Associate/Professional).
- Experience with containerization (Docker, Kubernetes) and orchestration tools.
- Knowledge of real-time data streaming using Kafka, Event Hubs, or similar technologies.
- Exposure to Machine Learning (ML) workflows in Databricks is a plus.

- Competitive Salary
- Performance based Variable Pay
- Health, Vision, Dental, and Life Insurance
- Comprehensive Employee Wellness Plan
- Retirement plan
- Paid Holidays and Vacation Time Accrual
- Encouraging and highly energetic setting
- Personal and Professional Growth Opportunities

📌 Data and Analytics - Lead (India)
🏢 Orion India Systems Private
📍 India