Job Title: Data Engineer - Databricks (AWS) | PySpark | Delta Lake
Location: Bengaluru, Noida, Gandhi Nagar, Pune, Chennai
Experience: 4+ Years

Job Summary
We are looking for a talented Data Engineer with strong expertise in Databricks on AWS, PySpark, Delta Lake, and contemporary data engineering practices. The ideal candidate will be responsible for building scalable data pipelines, implementing data governance frameworks, optimizing platform performance, and supporting enterprise analytics initiatives within the Healthcare/Life Sciences domain.

Key Responsibilities
Design, develop, and maintain scalable data pipelines using Databricks on AWS.
Build and support batch and streaming data ingestion frameworks.
Implement data quality, governance, lineage, and security controls using Unity Catalog and Databricks best practices.
Develop and optimize ETL/ELT processes using PySpark and SQL.
Design and implement Medallion Architecture (Bronze, Silver, Gold) data solutions.
Build and maintain CDC-based ingestion pipelines using AWS and Databricks services.
Optimize Spark performance,



cluster utilization, and platform costs.
Implement and support Delta Lake features to ensure data reliability and governance.
Manage Structured Streaming, Auto Loader, and real-time data processing requirements.
Develop CI/CD processes using Git, Databricks Asset Bundles, and deployment automation tools.
Monitor production environments, troubleshoot issues, and ensure SLA compliance.
Collaborate with business stakeholders, analysts, and engineering teams to deliver high-quality data solutions.
Mentor junior engineers and contribute to data engineering best practices.

Required Skills & Qualifications
Technical Skills
4+ years of experience in Data Engineering.
4+ years of hands-on experience with Databricks on AWS.
Strong expertise in PySpark and Spark optimization techniques.
Experience with Delta Lake and Lakehouse architecture.
Strong SQL programming skills.
Experience with Unity Catalog and Da

📌 AWS Data Engineer (Gandhinagar)
🏢 Tata Consultancy Services
📍 Gandhinagar