Aws Data Engineer (Hyderabad)

Aws Data Engineer (Hyderabad)

17 Nov
|
CGI
|
Hyderabad

17 Nov

CGI

Hyderabad

Role & responsibilities

- Develop and maintain scalable ETL/ELT pipelines using AWS Glue, Databricks, and PySpark to process large volumes of data.
- Schedule and monitor ETL pipelines using AWS Glue Workflows, Databricks Workflows, or Apache Airflow.
- Write Python scripts for data ingestion, transformation, automation, and error handling.
- Use PySpark DataFrames, Spark SQL, joins, aggregations, and window functions to transform large datasets.
- Write complex SQL queries using joins, CTEs, subqueries, window functions, and aggregations.
- Work with AWS services such as Amazon S3, AWS Glue, Lambda, IAM, Redshift, and CloudWatch.
- Develop Databricks notebooks and manage jobs, clusters or compute, and Delta Lake tables.
- Implement incremental data loads, change data capture (CDC), and data processing workflows.
- Optimize Spark jobs, partitioning, joins, caching, and shuffle operations to improve performance and reduce cloud costs.
- Optimize SQL queries, table design, and file formats such as Parquet and Delta Lake for effective data processing.
- Implement data quality checks to identify null values, duplicate records, and data inconsistencies.
- Troubleshoot ETL failures,



analyze execution logs, and resolve pipeline performance issues.
- Implement error handling, retry mechanisms, logging, and alerting for reliable pipeline execution.
- Apply AWS IAM roles, access controls, and encryption to maintain data security.
- Use Git for version control and collaborate with team members through code reviews and deployment workflows.
- Monitor pipeline execution, resource utilization, job failures, and processing costs using AWS CloudWatch and Databricks monitoring tools.

Preferred candidate profile

- Strong knowledge of Python programming, including functions, data structures, exception handling, and file processing.
- Proficiency in SQL, including joins, CTEs, subqueries, window functions, and query optimization.
- Hands-on experience with PySpark for large-scale data processing, transformations, joins, and aggregations.
- Experience with Databricks notebooks, jobs, Delta Lake, and data processing workflows.
- Strong understanding of AWS services, including Amazon S3, AWS Glue, Lambda, IAM, Redshift, and CloudWatch.

📌 Aws Data Engineer (Hyderabad)
🏢 CGI
📍 Hyderabad

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: aws data engineer (hyderabad) / hyderabad

Subscribe to this job alert:

Get the latest job offers by email for: aws data engineer (hyderabad) / hyderabad