Data Engineer- ETL (Hyderabad)

Data Engineer- ETL (Hyderabad)

17 Nov
|
CGI
|
Hyderabad

17 Nov

CGI

Hyderabad

Role & responsibilities

- Design, develop, and maintain scalable ETL/ELT pipelines to extract, transform, and load data from multiple sources.
- Develop Python scripts for data ingestion, transformation, automation, and data processing.
- Use PySpark to process large datasets, perform distributed transformations, joins, aggregations, and data cleansing.
- Develop and maintain Databricks notebooks, jobs, and data processing workflows.
- Write complex SQL queries using joins, CTEs, subqueries, window functions, and aggregations.
- Build and optimize data pipelines to ensure data accuracy, consistency, and reliability.
- Implement incremental data loading, deduplication, and change data capture (CDC) where required.
- Optimize Spark jobs, partitioning, caching, joins, and shuffle operations to improve performance.
- Optimize SQL queries and data processing logic to reduce execution time and resource consumption.
- Work with data formats such as CSV, JSON, and Parquet, and manage Delta Lake tables.
- Implement data quality checks, validation rules, and reconciliation processes.




- Schedule and monitor ETL workflows using Databricks Workflows or orchestration tools such as Apache Airflow.

Preferred candidate profile

- Strong knowledge of ETL/ELT concepts, data integration, data extraction, transformation, and loading processes.
- Proficiency in Python programming, including functions, data structures, exception handling, and file processing.
- Hands-on experience with PySpark DataFrames, transformations, actions, joins, aggregations, and window functions.
- Experience with Databricks notebooks, jobs, clusters or compute, and Delta Lake.
- Robust SQL skills, including joins, CTEs, subqueries, window functions, aggregations, and query optimization.
- Understanding of Apache Spark architecture, lazy evaluation, partitioning, caching, shuffling, and performance tuning.
- Knowledge of data formats such as CSV, JSON, Avro, and Parquet, along with Delta Lake tables.

📌 Data Engineer- ETL (Hyderabad)
🏢 CGI
📍 Hyderabad

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: data engineer- etl (hyderabad) / hyderabad

Subscribe to this job alert:

Get the latest job offers by email for: data engineer- etl (hyderabad) / hyderabad