Senior Databricks Engineer | 5+ Years | Hybrid or Remote (India) | Immediate Joiners Only (0–15 days) Are you a Databricks expert who loves building fast, reliable data pipelines? Join Xpertnest and help us build our modern Lakehouse platform, with hands-on work on Databricks ETL, Delta Lake and Databricks AI tools such as AI/BI Genie.

About the role
We are hiring a Senior Databricks Engineer to design, build and run scalable data pipelines on the Databricks Lakehouse Platform. You will own ETL/ELT workloads end to end, from ingestion through to curated, analytics-ready data. You will work closely with data architects, analysts, data scientists and business teams across locations.

Key responsibilities
• Design, develop and maintain ETL/ELT pipelines on Databricks using PySpark, Spark SQL and Scala.
• Build and manage Delta Lake tables using the Medallion architecture (Bronze, Silver, Gold).
• Develop batch and streaming pipelines with Structured Streaming, Auto Loader and Delta Live Tables (DLT).
• Orchestrate workloads with Databricks Workflows/Jobs, and integrate with tools such as Azure Data Factory or Airflow.
• Implement data governance, access control and lineage using Unity Catalog.
• Tune Spark jobs and clusters for performance and cost (partitioning, Z-ordering, liquid clustering, caching, Photon).
• Ingest data from databases, APIs, files and cloud storage (ADLS, S3, GCS) into the Lakehouse.
• Apply data quality checks, monitoring, alerting and error handling across pipelines.
• Set up CI/CD for Databricks code using Git,



Databricks Asset Bundles and Azure DevOps or GitHub Actions.
• Review code, mentor junior engineers and set engineering best practices.
• Work with stakeholders to turn business requirements into technical designs and documentation. Required skills and experience
• 5+ years of hands-on experience building production data pipelines on Databricks.
• Strong expertise in Databricks ETL: notebooks, Jobs/Workflows, Delta Live Tables and Auto Loader.
• Advanced PySpark and Spark SQL; solid understanding of Spark internals and performance tuning.
• Deep working knowledge of Delta Lake (ACID transactions, time travel, MERGE/upserts, schema evolution, OPTIMIZE/VACUUM).
• Hands-on experience with Unity Catalog for governance and security.
• Strong Python and SQL skills.
• Experience on at least one cloud: Azure (preferred), AWS or GCP, including its storage and security services.
• Good grasp of data modelling (dimensional, star/snowflake schemas) and data warehousing concepts.
• Experience with Git-based version control and CI/CD pipelines.
• Bachelor's degree in Computer Science, Engineering or a related field.




• Clear communication and the ability to work independently in a fast-paced team. Good to have
• Knowledge of Databricks AI tools (a strong plus):
• AI/BI Genie (Genie spaces for natural-language querying of data)
• Databricks Assistant for code, SQL and notebook help
• Mosaic AI: Model Serving, Vector Search, AI Functions (e.g. ai_query) and Agent Framework
• Building RAG or GenAI applications on the Lakehouse
• Databricks certifications: Data Engineer Associate or Professional; Generative AI Engineer Associate is a plus.
• Experience with Databricks SQL, dashboards and serverless compute.
• Exposure to MLflow, Feature Store or supporting ML workloads.
• Streaming experience with Kafka, Azure Event Hubs or Kinesis.
• Infrastructure as Code with Terraform.
• Experience migrating from legacy platforms (Hadoop, Informatica, SSIS, on-prem warehouses) to Databricks.
• Familiarity with Power BI, Tableau or other BI tools.
• Experience in an Agile/Scrum team. Work arrangement and joining
• Hybrid or remote: this is a full time role open as hybrid or fully remote from anywhere in India. Remote team members should be available for overlap with defined hours (and for occasional in-person meetings.
• Immediate joiners only: we are looking for candidates who can join within 0–15 days. Candidates serving notice with a confirmed last working day within this window are welcome to apply.
• Please do not apply if you have a notice period longer than 15 days.

📌 Senior Databricks Expert (India)
🏢 XpertNest
📍 India