Key Responsibilities

- Design, develop, and maintain large-scale data processing pipelines using PySpark and AWS services.
- Build and optimize Spark-based ETL solutions for enterprise data platforms.
- Work with AWS services including EMR, S3, IAM, Lambda, SNS, and SQS.
- Develop efficient data transformation and processing logic using Python and PySpark.
- Analyze and optimize Spark jobs using Spark UI, debugging techniques, and performance tuning methodologies.
- Write and optimize SQL queries involving joins, subqueries, CTEs, and complex transformations.
- Support data integration, ingestion, and reporting requirements across multiple systems.
- Participate in solution design using Data Vault, Data Mesh, and Data Fabric architectural concepts.
- Collaborate with business users, architects, and technical teams to understand requirements and deliver scalable solutions.
- Participate in Agile ceremonies and contribute to project planning, estimation, and execution.
- Troubleshoot production issues and implement performance improvements across data platforms.

Required Skills

- 4-7 years of experience in Data Engineering or Big Data development.




- Strong hands-on experience with PySpark and Apache Spark.
- Experience working with AWS EMR and AWS cloud services.
- Strong Python programming and scripting skills.
- Experience with AWS S3, IAM, Lambda, SNS, and SQS.
- Valuable understanding of Big Data architecture and distributed computing.
- Strong SQL skills including joins, subqueries, and CTEs.
- Experience with database technologies and data processing frameworks.
- Knowledge of Spark optimization, debugging, and performance tuning.

Preferred Skills

- Understanding of Data Vault architecture.
- Knowledge of Data Mesh and Data Fabric concepts.
- Experience with large-scale cloud data migration projects.
- Exposure to Hadoop and related Big Data ecosystems.
- Agile/Scrum project experience.

Desired Candidate Profile

- Robust analytical and problem-solving skills.
- Excellent communication and stakeholder management abilities.
- Experience working directly with business and IT teams.
- Ability to estimate effort, plan work, and deliver projects effectively.
- Self-motivated, proactive, and adaptable to fast-paced environments.

📌 AWS Big Data Engineer (PySpark & EMR) (Indore)
🏢 Tata Consultancy Services
📍 Indore