We are Hiring Technical Lead Data Engineering | Kafka, AWS, Spark (Scala/PySpark), Kubernetes/EKS

Company: Coforge Ltd.

Job Location: Hyderabad, Pune & Greater Noida Only.

Experience: 8 to 12 Years

Employment Type: Full-Time

Key Skills:- Kafka Must Have, Spark, Scala, PySpark, AWS, S3, Glue, EMR, Lambda, Kinesis, Athena, Redshift, EKS, Kubernetes, Docker, Helm, Python, SQL, Terraform, CI/CD, ETL, ELT, Real-Time Data Pipelines, Data Engineering, Technical Leadership.

Share your CV: [email protected]

If you have any questions regarding this chance, please feel free to contact us on WhatsApp at: (phone hidden) ( Gaurav Kumar HR, Coforge Ltd )

Role Overview:-

We are seeking an experienced Technical Lead Data Engineering to lead a team of Data Engineers in designing, developing, and supporting large-scale, cloud-native data platforms.

The ideal candidate will possess strong expertise in Kafka, AWS Data Engineering services, Spark (Scala/PySpark), and Kubernetes/EKS, along with proven experience in technical leadership, architecture, and stakeholder management.

The role requires hands-on technical leadership while driving engineering best practices, mentoring team members, and ensuring successful delivery of scalable, secure, and high-performance data solutions.

Key Responsibilities:-

Technical Leadership

- Lead and mentor a team of Data Engineers across design, development, testing, and production support.
- Drive architecture and technical design decisions for enterprise-scale data platforms.
- Establish engineering standards, coding practices, CI/CD processes, and operational excellence.
- Conduct code reviews and provide technical guidance to team members.
- Collaborate with architects,



product owners, and business stakeholders to translate requirements into scalable solutions.

Data Engineering & Platform Development

- Design and develop real-time and batch data pipelines using Kafka, Spark, Scala, and PySpark.
- Build scalable data processing frameworks on AWS.
- Develop and optimise ETL/ELT pipelines for high-volume structured and unstructured datasets.
- Ensure data quality, governance, observability, and operational monitoring.
- Troubleshoot performance bottlenecks and optimise data processing workloads.

Cloud & Kubernetes

- Deploy and manage data workloads on Amazon EKS (Elastic Kubernetes Service).
- Implement containerised applications using Docker and Kubernetes.
- Develop infrastructure automation and deployment pipelines.
- Work closely with DevOps teams to implement CI/CD and infrastructure-as-code practices.

Stakeholder & Delivery Management:-

- Participate in sprint planning, estimation, and technical roadmaps.
- Manage technical risks, dependencies, and delivery commitments.
- Provide status updates to leadership and stakeholders.
- Support production issues and critical incident resolution.

Required Skills

Data Engineering:-

- Strong experience with Apache Kafka including producers, consumers, Kafka Streams, schema management, and event-driven architectures.




- Expertise in Apache Spark using Scala and PySpark.
- Experience designing and supporting real-time and batch data processing solutions.
- Strong understanding of distributed systems and big data concepts.

AWS Data Engineering - Hands-on experience with:

- AWS S3
- AWS Glue
- AWS EMR
- AWS Lambda
- AWS Kinesis
- AWS Athena
- AWS Redshift
- AWS IAM
- CloudWatch
- Step Functions

Kubernetes & Containers:

- Strong expertise in Amazon EKS
- Kubernetes administration and deployment strategies
- Docker containerisation
- Helm Charts
- Autoscaling and resource optimisation

Programming:

- Scala
- Python
- SQL
- Shell Scripting

DevOps:

- Git/GitHub/GitLab
- Jenkins/Azure DevOps/GitHub Actions
- Terraform (preferred)
- CI/CD pipelines

Leadership Expectations:-

- Lead distributed teams and provide technical mentorship.
- Drive solution design discussions and architecture reviews.
- Ensure adherence to security, compliance, and operational standards.
- Foster innovation and continuous improvement within the team.
- Act as the primary technical escalation point for critical issues.

Preferred Qualifications:-

- Experience working in Banking/Financial Services environments.
- Exposure to Data Lakehouse architectures.
- Experience with Delta Lake, Iceberg, or Hudi.
- Knowledge of data governance and data quality frameworks.
- AWS Certifications (Solutions Architect, Data Engineer, Developer Associate).
- Kubernetes Certification (CKA/CKAD) is a plus.

Join This WhatsApp Group for Regular and Quick Job Updates.

Click On the Link:-

https://chat.whatsapp.com/LW8Dnu6IZ54HzkZVMakCE7

📌 Hiring Technical Lead Data Engineering | Kafka, AWS, PySpark, Scala (Hyderabad)
🏢 Coforge
📍 Hyderabad