AWS Data Engineer / Databricks / PySpark Technical Lead

Experience: 5-10 Years

Location: Bangalore (Whitefield)

Job Summary

We are seeking an experienced AWS Data Engineer / Databricks / PySpark Technical Lead with strong expertise in designing, developing, and implementing large-scale data solutions on AWS and Databricks platforms. The ideal candidate should have extensive hands-on experience with Python, PySpark, Spark SQL, Kafka, Airflow, and cloud-based data engineering solutions.

Key Responsibilities

- Design, develop, and optimize scalable data pipelines using Python, PySpark, Spark SQL, Databricks, and AWS Services. - Lead end-to-end architecture, design, development, and implementation of big data solutions. - Develop and maintain real-time and batch data processing workflows using Apache Kafka and Airflow. - Build and support data integration solutions using APIs, streaming platforms, and cloud-native services. - Perform data modeling, database design, performance tuning, and optimization activities. - Support production systems, troubleshoot critical issues, and provide timely resolutions. - Collaborate with business stakeholders, cross-functional teams, and vendors to deliver high-quality solutions. - Mentor junior team members and provide technical leadership in Agile development environments.



- Participate in code reviews, architecture discussions, and best practice implementation.

Required Skills

- Solid experience in Python, PySpark, Spark SQL, Databricks, AWS Cloud Services. - Minimum 2+ years of Apache Kafka streaming experience. - Hands-on experience with Apache Airflow. - Experience working with Oracle, PostgreSQL, and NoSQL databases. - Good knowledge of SQL, PL/SQL, Unix Shell Scripting, and Autosys. - Experience with GitHub, Agile methodologies, and CI/CD practices. - Strong analytical, troubleshooting, and problem-solving skills. - Excellent communication and stakeholder management skills.

Preferred Qualifications

- Experience in Banking or Capital Markets domain. - Proven ability to lead Agile development teams and deliver projects within tight deadlines. - Experience in designing enterprise-scale data platforms and cloud migration initiatives.

Note: This role requires approximately 90% hands-on development, design, and architecture work, along with 10% team mentoring and technical leadership responsibilities.

Provide your feedback on BizChat

Make this JD more conciseAdd key performance indicators for this roleCreate interview questions based on this JD

📌 AWS Data Engineer/ Data Bricks/Pyspark (Bengaluru)
🏢 Tata Consultancy Services
📍 Bengaluru