Job Summary

Experience: 5-8 years.

Responsibilities

- Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, SQL, and Delta Lake.

- Build and optimize batch and real-time data pipelines on AWS using services such as S3, Lambda, Glue, EMR, Redshift, and related cloud-native components.

- Develop, implement, and support Delta Live Tables (DLT) pipelines for reliable, high-quality, and auditable data processing.

- Work with Unity Catalog to implement data governance, access control, lineage, and security best practices.

- Develop and optimize ETL/ELT pipelines with a focus on performance tuning, scalability, reliability, and cost efficiency.

- Implement infrastructure and platform automation using Terraform and Infrastructure as Code (IaC) principles.

- Build and maintain CI/CD pipelines using Jenkins, Git, and related DevOps tools to enable automated deployments and testing.

- Support deployment, monitoring, and maintenance of Databricks workloads across development, testing, and production environments.

- Collaborate with architects, product owners, data scientists, and business stakeholders to understand requirements and deliver high-quality solutions.

- Participate in cloud migration, modernization, and data platform transformation initiatives on AWS.

- Contribute to data quality, observability, logging, monitoring, and alerting frameworks to ensure platform reliability.

- Work with APIs, data integration frameworks, and external systems to enable seamless data exchange across platforms.

- Troubleshoot production issues, perform root cause analysis, and drive continuous improvement initiatives.

- Follow software engineering best practices, participate in code reviews, and ensure adherence to coding, security, and operational standards.

- Work effectively within Agile teams, contributing to sprint planning,



estimation, development, and release activities.

- Collaborate with global teams and support business-critical deliveries across different time zones when required.

Qualifications

Profile Required

Senior-level Software Engineer with deep Databricks, AWS, and Python data platform expertise, capable of designing and leading enterprise-scale AI-enabled data solutions.

Minimum Qualifications

- Deep hands-on experience with Databricks (PySpark, SQL, Delta Lake) in large-scale production environments.

- Proven experience designing and leading end-to-end data pipelines on Databricks.

- Strong hands-on experience with Delta Live Tables (DLT) for governed data processing.

- Advanced experience with Unity Catalog for data governance, access control, lineage, and metadata.

- Expertise in Spark performance tuning, scalability, and cost optimization.

- Experience managing Databricks workflows, monitoring, and production operations.

- Strong knowledge of cloud-based data platforms (AWS preferred).

- Proven ability to mentor engineers and lead technical delivery.

Preferred Qualifications

- Experience building and operating AI agents for workflow automation and decision support.

- Robust understanding of data observability, platform reliability, and FinOps practices in Databricks.

- Experience working in regulated or financial services domains.

- Exposure to enterprise-scale AI governance, compliance, and audit requirements.

- Ability to influence architecture decisions and align stakeholders.

- Excellent communication skills with both technical and non-technical audiences.

Job Type

Permanent

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Specialist Software Engineer - Python + Spark - (26000 (Bengaluru)
🏢 SA Technologies
📍 Bengaluru