Sr. Databricks Engineer- 20823

We are looking for an experienced and technically strong Senior Databricks Developer with 5–8 years of experience in PySpark and Databricks. This role involves leading the design and development of scalable data pipelines, reusable accelerator components, and performance-optimized Spark workloads. The candidate will work closely with architects and platform teams and take ownership of delivering production-grade data solutions.

Key Responsibilities:

Lead the design and development of scalable, reusable data pipelines and accelerator frameworks using PySpark and Databricks.

Design and execute testing strategies for both structured and unstructured data (PDFs, Text files, and RTF documents) to ensure high-fidelity transformation into structured formats within the Data Lakehouse.

Validate all Source-to-Target (S2T) mappings across bronze, silver, and gold layers to ensure data lineage and integrity.

Collaborate with architects and stakeholders to translate business and technical requirements into robust data solutions.

Own end-to-end development,



testing, and deployment of data pipelines across multiple environments.

Drive Spark performance optimization, cost efficiency, and best practices across Databricks workloads.

Design and manage workflow orchestration using Databricks Workflows and Apache Airflow.

Leverage strong PYSpark/ Spark SQL knowledge to create mockup data for exhaustive edge-case testing.

Review code, mentor junior developers, and enforce engineering best practices.

Monitor, troubleshoot, and resolve complex development issues to ensure SLA and data reliability.

Contribute to CI/CD pipeline design and automation for data engineering workflows.

Required Skills & Experience:

5–8 years of experience in data engineering with strong expertise in PySpark and distributed data processing.

Extensive hands-on experience with Databricks (Notebooks, Jobs, Workflows, Delta Lake).

Robust command of Spark SQL, Advanced SQ

📌 Sr. Databricks Engineer (Chennai)
🏢 TalentOla
📍 Chennai