CGI is seeking an experienced Python, AWS Services, Pyspark Developer to design, develop, optimize, and maintain scalable data engineering solutions on cloud platforms. The ideal candidate will have solid hands-on experience in Python programming, distributed data processing using PySpark, and building data pipelines using AWS services.
The candidate will be responsible for developing robust ETL/ELT pipelines, processing large volumes of structured and unstructured data, implementing data transformations, and ensuring data quality, reliability, and performance. The role requires close collaboration with Data Engineers, Data Architects, Business Analysts, and cross-functional teams to deliver enterprise-grade data solutions.
Your future duties and responsibilities
• Design, develop, test, and maintain scalable data processing applications using Python and PySpark.
• Build and maintain ETL/ELT pipelines for data ingestion, transformation, validation, and integration.
• Develop distributed data processing solutions using Apache Spark and PySpark to process large datasets efficiently.
• Design, implement, and manage cloud-based data pipelines using AWS services.
• Work with AWS services such as Amazon S3, AWS Glue, AWS Lambda, Amazon EMR, Amazon Redshift, AWS Step Functions, and Amazon CloudWatch, as applicable to the project.
• Develop data extraction, transformation, and loading processes from multiple data sources into data lakes and data warehouses.
• Implement data transformation logic, business rules, aggregations, joins, and data cleansing using Python and PySpark.
• Optimize Spark jobs through partitioning, caching, join strategies, and resource tuning to improve performance and reduce processing time.
• Develop and maintain reusable Python modules, libraries, utilities, and automation scripts.
• Implement data validation, reconciliation, error handling, logging, and exception management within data pipelines.
• Monitor pipeline execution, troubleshoot failu
📌 Python, AWS Services, Pyspark Developer (Hyderabad)
🏢 CGI
📍 Hyderabad