Skill: Pyspark, AWS

Exp: 6 to 12 years

Location: Pune

Job Summary

We are seeking an experienced AWS PySpark Developer to design, develop, and optimize large-scale data processing pipelines in the cloud. The ideal candidate will have strong expertise in PySpark, AWS services, and ETL workflows, with a focus on building scalable, high-performance data solutions for analytics and reporting.

Key Responsibilities

- Design, develop, and maintain ETL pipelines using PySpark on AWS.
- Work with large datasets to perform data cleansing, transformation, and aggregation.
- Optimize Spark jobs for performance and cost efficiency.
- Integrate data from multiple sources including AWS S3, RDS, Redshift, DynamoDB, and APIs.
- Implement data quality checks and ensure compliance with governance policies.
- Collaborate with data analysts, data scientists, and business stakeholders to deliver data solutions.
- Deploy and monitor jobs using AWS Glue, EMR, Step Functions, or Lambda.
- Troubleshoot and resolve issues in production data pipelines.




- Maintain documentation for data flows, transformations, and architecture.

Required Skills & Qualifications

- 3–7 years of experience in data engineering or big data development.
- Strong proficiency in PySpark and Spark SQL.
- Hands-on experience with AWS services such as:
- S3, Glue, EMR, Lambda, Athena, Redshift, IAM

- Solid understanding of ETL concepts and data warehousing.
- Proficiency in Python for scripting and automation.
- Experience with version control (Git) and CI/CD pipelines.
- Robust problem-solving skills and ability to work in an agile environment.

Preferred Skills

- Experience with AWS Glue Studio and Glue Catalog.
- Knowledge of Airflow or other orchestration tools.
- Familiarity with streaming data using Kafka or Kinesis.
- Exposure to Terraform or CloudFormation for infrastructure as code.
- Understanding of data security and encryption in AWS.

📌 AWS, Pyspark (Chennai)
🏢 Cognizant
📍 Chennai