15 Nov
|
XpertNest
|
India
Senior Databricks Engineer | 5+ Years | Hybrid or Remote (India) | Immediate Joiners Only (0–15 days)
Are you a Databricks expert who loves building fast, reliable data pipelines? Join Xpertnest and help us build our modern Lakehouse platform, with hands-on work on Databricks ETL, Delta Lake and Databricks AI tools such as AI/BI Genie.
About the role
We are hiring a Senior Databricks Engineer to design, build and run scalable data pipelines on the Databricks Lakehouse Platform. You will own ETL/ELT workloads end to end, from ingestion through to curated, analytics-ready data. You will work closely with data architects, analysts, data scientists and business teams across locations.
Key responsibilities
- Design, develop and maintain ETL/ELT pipelines on Databricks using PySpark, Spark SQL and Scala.
- Build and manage Delta Lake tables using the Medallion architecture (Bronze, Silver, Gold).
- Develop batch and streaming pipelines with Structured Streaming, Auto Loader and Delta Live Tables (DLT).
- Orchestrate workloads with Databricks Workflows/Jobs, and integrate with tools such as Azure Data Factory or Airflow.
- Implement data governance, access control and lineage using Unity Catalog.
- Tune Spark jobs and clusters for performance and cost (partitioning, Z-ordering, liquid clustering, caching, Photon).
- Ingest data from databases, APIs, files and cloud storage (ADLS, S3, GCS) into the Lakehouse.
- Apply data quality checks, monitoring, alerting and error handling across pipelines.
- Set up CI/CD for Databricks code using Git, Databricks Asset Bundles and Azure DevOps or GitHub Actions.
- Review code, mentor junior engineers and set engineering best practices.
- Work with stakeholders to turn business requirements into technical designs and documentation.
Required skills and experience
- 5+ years of hands-on experience building production data pipelines on Databricks.
- Strong expertise in Databricks ETL: notebooks, Jobs/Workflows, Delta Live Tables and Auto Loader.
- Advanced PySpark and Spark SQL; solid understanding of Spark internals and performance tuning.
- Deep working knowledge of Delta Lake (ACID transactions, time travel, MERGE/upserts, schema evolution, OPTIMIZE/VACUUM).
- Hands-on experience with Unity Catalog for governance and security.
- Solid Python and SQL skills.
- Experience on at least one cloud: Azure (preferred), AWS or GCP, including its storage and security services.
- Good grasp of data modelling (dimensional, star/snowflake schemas) and data warehousing concepts.
- Experience with Git-based version control and CI/CD pipelines.
- Bachelor's degree in Computer Science, Engineering or a related field.
- Clear communication and the ability to work independently in a fast-paced team.
Good to have
- Knowledge of Databricks AI tools (a strong plus):
- AI/BI Genie (Genie spaces for natural-language querying of data)
- Databricks Assistant for code, SQL and notebook help
- Mosaic AI: Model Serving, Vector Search, AI Functions (e.g. ai_query) and Agent Framework
- Building RAG or GenAI applications on the Lakehouse
- Databricks certifications: Data Engineer Associate or Professional; Generative AI Engineer Associate is a plus.
- Experience with Databricks SQL, dashboards and serverless compute.
- Exposure to MLflow, Feature Store or supporting ML workloads.
- Streaming experience with Kafka, Azure Event Hubs or Kinesis.
- Infrastructure as Code with Terraform.
- Experience migrating from legacy platforms (Hadoop, Informatica, SSIS, on-prem warehouses) to Databricks.
- Familiarity with Power BI, Tableau or other BI tools.
- Experience in an Agile/Scrum team.
Work arrangement and joining
- Hybrid or remot e: this is a full-time role open as hybrid or fully remote from anywhere in India. Remote team members should be available for overlap with defined hours (and for occasional in-person meetings.
- Immediate joiners only: we are looking for candidates who can join within 0–15 days. Candidates serving notice with a confirmed last working day within this window are welcome to apply.
- Please do not apply if you have a notice period longer than 15 days.
📌 Senior Databricks Expert (India)
🏢 XpertNest
📍 India