- Job Title: Senior Data Engineer (Databricks)

Role Overview: We are seeking a Senior Data Engineer with deep expertise in Databricks, Apache Spark, and cloud platforms to design, build, and optimize enterprise-grade lakehouse data pipelines. In this role, you will implement Medallion architectures, develop custom REST APIs via FastAPI for downstream consumption, manage governance via Unity Catalog, and automate deployments using modern CI/CD practices.

Key Responsibilities:

- Design, build, and optimize scalable batch and streaming data pipelines using Databricks, PySpark, Spark SQL, and Delta Lake.

- Implement robust Medallion architectures (Bronze, Silver, Gold layers) with automated data quality checks and validation frameworks.

- Develop production-grade REST APIs using FastAPI to expose lakehouse data to downstream consumers and microservices.

- Enforce fine-grained data governance, access controls, and data lineage across data assets using Unity Catalog.

- Deploy, monitor, and manage production pipelines using Databricks Workflows, Jobs, and orchestration engines such as Azure Data Factory (ADF).

- Automate build and deployment pipelines using Databricks Asset Bundles (DABs), Git, and CI/CD frameworks (Azure DevOps or GitHub Actions).

- Tune Apache Spark jobs, cluster configurations, DAG execution, partitioning, and caching to ensure performance, reliability, and cost efficiency.

- Troubleshoot production issues,



implement alerting and logging frameworks, and collaborate with enterprise cross-functional teams.

Preferred Qualifications:

Databricks Certified Data Engineer (Associate or Professional).

Exposure to Lakebase Postgres, Databricks Genie coding agent, or multi-cloud environments (GCP).

Hands-on experience with Apache Airflow or event-streaming engines (Kafka, Event Hubs).

Required Qualifications:

5 to 7 years of overall data engineering experience, with 3+ years of dedicated hands-on experience in Databricks and Apache Spark (PySpark, Spark SQL).

Strong proficiency in Python, SQL, and advanced API development using FastAPI.

Hands-on experience with Delta Lake, Delta Live Tables / Lakeflow, Lakehouse Apps, and Databricks REST APIs.

Solid understanding of data governance, security, and cataloging using Unity Catalog.

Proven experience with Microsoft Azure data ecosystem, including Azure Data Factory (ADF) and ADLS Gen2.

Practical experience with up-to-date data DevOps: Git, CI/CD pipelines, and Databricks Asset Bundles (DABs).

Deep understanding of Spark compute, node engines, parallel processing, and DAG execution frameworks.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Data Engineers (Databricks) (Chennai)
🏢 Ibu
📍 Chennai