? Work Mode: Remote

? Employment Type: Part-Time | Freelance

About the Opportunity

We are looking for a experienced AWS EMR / Apache Spark Infrastructure Engineer to provide technical guidance and hands-on troubleshooting support for an enterprise-scale DRAM memory optimization project.

The ideal candidate will have strong expertise in Amazon EMR cluster architecture, Apache Spark memory management, distributed workload optimization, and AWS infrastructure right-sizing. You will help identify memory bottlenecks, optimize cluster configurations, evaluate AWS Graviton-based instances, and improve infrastructure efficiency while maintaining performance and reliability.

Key Responsibilities

- Analyze AWS EMR cluster architecture, resource allocation, and memory consumption.
- Optimize Apache Spark executor sizing, driver memory, memory overhead, and dynamic allocation.
- Troubleshoot OutOfMemoryError, executor failures, garbage collection issues, and memory spills.
- Evaluate AWS EC2 instance families, Graviton/ARM64 compatibility, and cluster right-sizing opportunities.
- Identify opportunities to reduce DRAM consumption and infrastructure costs across distributed workloads.
- Monitor Spark UI metrics, Amazon CloudWatch, EMR logs, CPU utilization, and executor performance.
- Design and execute controlled performance optimization experiments in non-production environments.
- Compare baseline and optimized configurations using measurable memory, throughput, latency, and cost metrics.




- Validate performance improvements without compromising workload reliability or availability.
- Provide technical guidance on configuration changes, deployment strategies, rollback procedures, and production-safe optimization.
- Prepare technical documentation, optimization recommendations, runbooks, and SOPs.

Required Technical Skills

- 5+ years of strong hands-on experience with Amazon EMR infrastructure and cluster optimization.
- Advanced understanding of Apache Spark architecture and memory management.
- Expertise in executor and driver memory tuning, memory overhead, garbage collection, and OOM troubleshooting.
- Experience with Spark Dynamic Allocation and executor scaling.
- Experience evaluating AWS Graviton / ARM64-based EC2 instances for distributed workloads.
- Strong knowledge of AWS EC2, S3, IAM, CloudWatch, and infrastructure monitoring.
- Proficiency in Spark UI, EMR logs, performance metrics, and root-cause analysis.
- Understanding of JVM memory management and large-scale distributed systems.
- Experience with PySpark, Spark SQL, Scala, or Java.
- Linux administration and shell scripting skills.

Preferred Qualifications

- Experience with enterprise-scale big data platforms and production Spark workloads.
- Infrastructure automation and configuration management experience.
- Proven ability to conduct performance benchmarking and infrastructure cost optimization.
- Solid analytical, troubleshooting, communication, and technical documentation skills.

📌 Freelance Senior AWS EMR / Spark Infrastructure Engineer (India)
🏢 IndiTech Consulting
📍 India