Job Title: Data Engineer - Databricks (AWS) | PySpark | Delta Lake

Location: Bengaluru, Noida, Gandhi Nagar, Pune, Chennai

Experience: 4+ Years

Job Summary

We are looking for a talented Data Engineer with strong expertise in Databricks on AWS, PySpark, Delta Lake, and up-to-date data engineering practices. The ideal candidate will be responsible for building scalable data pipelines, implementing data governance frameworks, optimizing platform performance, and supporting enterprise analytics initiatives within the Healthcare/Life Sciences domain.

Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks on AWS.
- Build and support batch and streaming data ingestion frameworks.
- Implement data quality, governance, lineage, and security controls using Unity Catalog and Databricks best practices.
- Develop and optimize ETL/ELT processes using PySpark and SQL.
- Design and implement Medallion Architecture (Bronze, Silver, Gold) data solutions.
- Build and maintain CDC-based ingestion pipelines using AWS and Databricks services.




- Optimize Spark performance, cluster utilization, and platform costs.
- Implement and support Delta Lake features to ensure data reliability and governance.
- Manage Structured Streaming, Auto Loader, and real-time data processing requirements.
- Develop CI/CD processes using Git, Databricks Asset Bundles, and deployment automation tools.
- Monitor production environments, troubleshoot issues, and ensure SLA compliance.
- Collaborate with business stakeholders, analysts, and engineering teams to deliver high-quality data solutions.
- Mentor junior engineers and contribute to data engineering best practices.

Required Skills & Qualifications

Technical Skills
- 4+ years of experience in Data Engineering.
- 4+ years of hands-on experience with Databricks on AWS.
- Strong expertise in PySpark and Spark optimization techniques.
- Experience with Delta Lake and Lakehouse architecture.
- Strong SQL programming skil

📌 AWS Data Engineer (Bhavnagar)
🏢 Tata Consultancy Services
📍 Bhavnagar