Job Title: Data Engineer - Databricks (AWS) | PySpark | Delta Lake

Location : Bengaluru, Noida, Gandhi Nagar, Pune, Chennai

Experience : 4+ Years Job Summary

We are looking for a talented Data Engineer with strong expertise in Databricks on AWS, PySpark, Delta Lake, and modern data engineering practices . The ideal candidate will be responsible for building scalable data pipelines, implementing data governance frameworks, optimizing platform performance, and supporting enterprise analytics initiatives within the Healthcare/Life Sciences domain. Key Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks on AWS.

Build and support batch and streaming data ingestion frameworks.

Implement data quality, governance, lineage, and security controls using Unity Catalog and Databricks best practices.

Develop and optimize ETL/ELT processes using PySpark and SQL.

Design and implement Medallion Architecture (Bronze, Silver, Gold) data solutions.

Build and maintain CDC-based ingestion pipelines using AWS and Databricks services.

Optimize Spark performance, cluster utilization, and platform costs.

Implement and support Delta Lake features to ensure data reliability and governance.

Manage Structured Streaming, Auto Loader, and real-time data processing requirements.

Develop CI/CD processes using Git, Databricks Asset Bundles, and deployment automation tools.

Monitor production environments, troubleshoot issues, and ensure SLA compliance.

Collaborate with business stakeholders, analysts,



and engineering teams to deliver high-quality data solutions.

Mentor junior engineers and contribute to data engineering best practices.

Required Skills & Qualifications

Technical Skills

4+ years of experience in Data Engineering.

4+ years of hands-on experience with Databricks on AWS.

Strong expertise in PySpark and Spark optimization techniques.

Experience with Delta Lake and Lakehouse architecture.

Strong SQL programming skills.

Experience with Unity Catalog and Databricks governance frameworks.

Knowledge of dimensional data modeling and Medallion Architecture.

Hands-on experience with:

Structured Streaming

Auto Loader

Change Data Capture (CDC)

Experience with AWS services

Amazon S3

IAM

AWS DMS

Kinesis and/or MSK

AWS Glue

Experience with Git, CI/CD pipelines, and Databricks Asset Bundles.

Experience supporting production environments, performance tuning, and cost optimization.

Domain Experience

Healthcare or Life Sciences domain experience preferred.

Understanding of data governance, analytics, privacy, and regulatory compliance requirements. Education

Bachelor's or Master's Degree in Computer Science, Information Technology, Engineering, or related field.

Flexible for candidates with strong relevant experience.

Certifications (Preferred)

Databricks Certified Data Engineer Associate/Professional

AWS Certified Data Engineer

AWS Certified Solutions Architect

Microsoft Azure Certified Developer

Google Skilled Cloud Developer

📌 AWS Data Engineer (Tirunelveli)
🏢 Tata Consultancy Services
📍 Tirunelveli