Skill: Pyspark, AWS

Exp: 6 to 12 years

Location: Pune

Job Summary

We are seeking an experienced AWS PySpark Developer to design, develop, and optimize large-scale data processing pipelines in the cloud. The ideal candidate will have robust expertise in PySpark, AWS services, and ETL workflows, with a focus on building scalable, high-performance data solutions for analytics and reporting.

Key Responsibilities

- Design, develop, and maintain ETL pipelines using PySpark on AWS.

- Work with large datasets to perform data cleansing, transformation, and aggregation.

- Optimize Spark jobs for performance and cost efficiency.

- Integrate data from multiple sources including AWS S3, RDS, Redshift, DynamoDB, and APIs.

- Implement data quality checks and ensure compliance with governance policies.

- Collaborate with data analysts, data scientists, and business stakeholders to deliver data solutions.

- Deploy and monitor jobs using AWS Glue, EMR, Step Functions, or Lambda.

- Troubleshoot and resolve issues in production data pipelines.





- Maintain documentation for data flows, transformations, and architecture.

Required Skills & Qualifications

- 3–7 years of experience in data engineering or big data development.

- Strong proficiency in PySpark and Spark SQL.

- Hands-on experience with AWS services such as:

- S3, Glue, EMR, Lambda, Athena, Redshift, IAM

- Solid understanding of ETL concepts and data warehousing.

- Proficiency in Python for scripting and automation.

- Experience with version control (Git) and CI/CD pipelines.

- Strong problem-solving skills and ability to work in an agile environment.

Preferred Skills

- Experience with AWS Glue Studio and Glue Catalog.

- Knowledge of Airflow or other orchestration tools.

- Familiarity with streaming data using Kafka or Kinesis.

- Exposure to Terraform or CloudFormation for infrastructure as code.

- Understanding of data security and encryption in AWS.

📌 AWS, Pyspark (Chennai)
🏢 Cognizant Consulting
📍 Chennai