AWS Data Engineer / Databricks / PySpark Technical Lead

Experience: 5-10 Years

Location: Bangalore (Whitefield)

Job Summary

We are seeking an experienced AWS Data Engineer / Databricks / PySpark Technical Lead with strong expertise in designing, developing, and implementing large-scale data solutions on AWS and Databricks platforms. The ideal candidate should have extensive hands-on experience with Python, PySpark, Spark SQL, Kafka, Airflow, and cloud-based data engineering solutions.

Key Responsibilities

- Design, develop, and optimize scalable data pipelines using Python, PySpark, Spark SQL, Databricks, and AWS Services.
- Lead end-to-end architecture, design, development, and implementation of big data solutions.
- Develop and maintain real-time and batch data processing workflows using Apache Kafka and Airflow.
- Build and support data integration solutions using APIs, streaming platforms, and cloud-native services.
- Perform data modeling, database design, performance tuning, and optimization activities.
- Support production systems, troubleshoot critical issues, and provide timely resolutions.
- Collaborate with business stakeholders, cross-functional teams, and vendors to deliver high-quality solutions.
- Mentor junior team members and provide technical leadership in Agile development environments.




- Participate in code reviews, architecture discussions, and best practice implementation.

Required Skills
- Strong experience in Python, PySpark, Spark SQL, Databricks, AWS Cloud Services.
- Minimum 2+ years of Apache Kafka streaming experience.
- Hands-on experience with Apache Airflow.
- Experience working with Oracle, PostgreSQL, and NoSQL databases.
- Good knowledge of SQL, PL/SQL, Unix Shell Scripting, and Autosys.
- Experience with GitHub, Agile methodologies, and CI/CD practices.
- Solid analytical, troubleshooting, and problem-solving skills.
- Excellent communication and stakeholder management skills.

Preferred Qualifications
- Experience in Banking or Capital Markets domain.
- Proven ability to lead Agile development teams and deliver projects within tight deadlines.
- Experience in designing enterprise-scale data platforms and cloud migration initiatives.

Note: This role requires approximately 90% hands-on development, design, and architecture work, along with 10% team mentoring and technical leadership responsibilities. Provide your feedback on BizChat

Make this JD more conciseAdd key performance indicators for this roleCreate interview questions based on this JD

📌 AWS Data Engineer/ Data Bricks/Pyspark (Bengaluru)
🏢 Tata Consultancy Services
📍 Bengaluru