We are is seeking a skilled and experienced Lead Databricks Engineer to join our dynamic team. The ideal candidate will have a strong background in Databricks, Pyspark, CI/CD practices, and MLOps. This role will be pivotal in developing data solutions that drive business insights and enhance operational efficiency. Locations: Bangalore, Pune and Hyderabad (Hybrid model)
Key Responsibilities
- Lead the design and implementation of Databricks-based data solutions.
- Develop, test, and deploy data processing pipelines using Pyspark.
- Collaborate with data scientists and analysts to implement MLOps practices for deploying machine learning models.
- Establish and maintain CI/CD pipelines to automate the deployment process and ensure high-quality code delivery.
- Optimize data workflows and improve system performance within the Databricks environment.
- Mentor and guide junior engineers and team members in best practices and emerging technologies.
- Work closely with stakeholders to understand business requirements and translate them into technical specifications. Required
Skills:
- Proficient in Databricks and its ecosystem.
- Robust experience with Pyspark for data processing and analysis.
- Solid understanding of CI/CD practices and tools.
- Familiarity with MLOps concepts and their application in real-world scenarios.
- Excellent problem-solving skills and attention to detail.
- Ability to work collaboratively in a fast-paced, team-oriented environment.
Qualifications:
- Bachelor’s degree in Computer Science, Engineering, or a related field.
- 8+years of experience in data engineering or a related role, with at least 2 years of experience specifically in Databricks.
- Proven experience in leveraging Pyspark for data processing.
- Experience implementing CI/CD practices in data projects.
- Knowledge of machine learning concepts and MLOps practices is a plus.