EKS & Spark Performance Optimization Engineer

Experience: 6-8 Years

Location: PAN India

We are seeking a highly skilled EKS & Spark Performance Optimization Engineer with strong expertise in AWS cloud technologies, Amazon EKS, EMR, and Apache Spark performance tuning. The ideal candidate will be responsible for optimizing large-scale data processing workloads, improving cluster efficiency, reducing execution costs, and enhancing application performance in cloud-native environments.

Key Responsibilities

- Design, implement, and optimize big data processing solutions on AWS.
- Manage and optimize workloads running on Amazon EKS and Amazon EMR.
- Perform Spark application performance tuning, troubleshooting, and capacity planning.
- Analyze Spark jobs and optimize resource allocation, memory management, executors, and partitioning strategies.
- Enhance cluster utilization, scalability, reliability, and cost efficiency.
- Monitor and troubleshoot production workloads running on Kubernetes and EMR environments.
- Implement best practices for containerization, orchestration, and cloud-native deployments.
- Collaborate with data engineers, architects, and platform teams to improve application performance.
- Optimize AWS infrastructure for high-performance distributed computing workloads.
- Create operational dashboards, monitoring solutions, and performance reports.





Required Skills

Core Skills

- Strong hands-on experience with Amazon EKS (Elastic Kubernetes Service).
- Expertise in Apache Spark performance tuning and optimization.
- Experience with Amazon EMR and large-scale data processing workloads.
- Solid understanding of distributed computing concepts and cluster management.
- Experience troubleshooting Spark job failures, memory issues, and performance bottlenecks.

AWS Technologies
- Amazon EKS
- Amazon EMR
- EC2
- S3
- CloudWatch
- IAM
- VPC
- AWS Cost Optimization

Data Engineering & Performance Tuning
- Spark SQL Optimization
- Executor and Driver Tuning
- Partitioning and Caching Strategies
- Shuffle Optimization
- Resource Allocation and Cluster Sizing
- Job Monitoring and Troubleshooting
- JVM Performance Tuning
- Performance Benchmarking

Container & DevOps
- Kubernetes
- Docker
- CI/CD Pipelines
- Infrastructure as Code (Terraform/CloudFormation preferred)
- Monitoring and Observability Tools

Preferred Skills
- Experience with Databricks or AWS Glue.
- Exposure to Hadoop ecosystem components.
- Knowledge of Airflow or other workflow orchestration tools.
- Experience with FinOps and cloud cost optimization initiatives.
- Understanding of Data Lake architectures and modern analytics platforms.

📌 EKS with Spark Tuning (Bengaluru)
🏢 Tata Consultancy Services
📍 Bengaluru