Experience: 8–10 Years

Location: Remote

Employment Type: Full-Time

Job Summary

We are seeking a highly experienced Senior Databricks Engineer to design, build, and optimize large-scale data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate will have deep expertise in big data processing, cloud architecture, and end-to-end data pipeline development, along with robust leadership skills to guide technical teams.

Key Responsibilities

- Design, develop, and optimize scalable data pipelines and ETL/ELT workflows using Databricks (PySpark, Spark SQL, Delta Lake)
- Architect and implement Lakehouse solutions integrating data lakes and data warehouses
- Lead the migration of legacy data platforms to Databricks on Azure/AWS/GCP
- Optimize Spark jobs for performance, cost, and scalability
- Implement data governance, security, and access control using Unity Catalog
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements
- Design and manage CI/CD pipelines for Databricks notebooks and jobs
- Mentor junior engineers and provide technical leadership on best practices
- Troubleshoot performance bottlenecks and ensure high availability of data pipelines
- Work with structured, semi-structured, and unstructured data at scale





Required Skills & Qualifications

- 8–10 years of overall experience in data engineering/big data, with strong hands-on expertise in Databricks
- Proficiency in PySpark, Spark SQL, and Delta Lake
- Strong experience with at least one major cloud platform (Azure Databricks, AWS Databricks, or GCP)
- Solid understanding of data warehousing concepts, data modeling, and ETL/ELT design patterns
- Experience with Unity Catalog, Delta Live Tables, and Databricks Workflows
- Proficiency in Python and SQL
- Knowledge of Azure Data Factory (ADF) and Medallion Architecture
- Experience with orchestration tools (Airflow, ADF, or similar)
- Familiarity with CI/CD tools (Git, Azure DevOps, Jenkins)
- Strong understanding of data governance, security, and compliance frameworks
- Excellent problem-solving, communication, and stakeholder management skills

Good to Have

- Databricks Certified Data Engineer Skilled/Associate certification
- Experience with MLflow and machine learning pipelines
- Knowledge of Kafka or other streaming technologies
- Experience working in the US healthcare industry is a plus

Education

- Bachelor's/Master's degree in Computer Science, Information Technology, or a related field

📌 Senior Databricks Engineer (India)
🏢 Quantorus
📍 India