We are looking for a Senior Databricks Engineer with 7+ years of experience in data engineering and strong hands-on expertise in Databricks, Apache Spark, and Claude Agent. The ideal candidate will design and build scalable data pipelines, optimize large-scale data processing, work with cloud-based data platforms, and leverage Claude Agent for AI-powered data engineering and workflow automation.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.

• Develop and optimize PySpark/Spark jobs for large-scale data processing.

• Build reliable ETL/ELT pipelines using Databricks workflows, Delta Lake, and SQL.

• Implement data transformations, data quality checks, and performance optimization.

• Work with cloud data platforms such as AWS, Azure, or GCP.

• Optimize Spark jobs, cluster configurations, partitioning, joins, and data processing performance.

• Design, develop, and integrate Claude Agent-based solutions for AI-powered data engineering, workflow automation, and intelligent data processing.





• Collaborate with data engineers, data scientists, analysts, and business teams to deliver data solutions.

• Implement best practices for CI/CD, version control, monitoring, and production support.

• Troubleshoot data pipeline and performance issues in production environments.

​

Required Skills

• 7+ years of experience in Data Engineering.

• Strong hands-on experience with Databricks.

• Solid expertise in Apache Spark / PySpark.

• Experience with Delta Lake, Databricks Workflows, and Spark SQL.

• Strong proficiency in Python and SQL.

• Experience building and optimizing large-scale ETL/ELT pipelines.

• Experience with at least one cloud platform: AWS, Azure, or GCP.

• Good understanding of data warehousing, data modeling, and distributed data processing.

• Experience with Git and CI/CD practices.

Preferred Skills

• Experience with Unity Catalog and Databricks governance.

• Experience with Azure

📌 Senior Databricks Engineer (Chennai)
🏢 Numentica
📍 Chennai