Role & responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines to extract, transform, and load data from multiple sources.
- Develop Python scripts for data ingestion, transformation, automation, and data processing.
- Use PySpark to process large datasets, perform distributed transformations, joins, aggregations, and data cleansing.
- Develop and maintain Databricks notebooks, jobs, and data processing workflows.
- Write complex SQL queries using joins, CTEs, subqueries, window functions, and aggregations.
- Build and optimize data pipelines to ensure data accuracy, consistency, and reliability.
- Implement incremental data loading, deduplication, and change data capture (CDC) where required.
- Optimize Spark jobs, partitioning, caching, joins, and shuffle operations to improve performance.
- Optimize SQL queries and data processing logic to reduce execution time and resource consumption.
- Work with data formats such as CSV, JSON, and Parquet, and manage Delta Lake tables.
- Implement data quality checks, validation rules, and reconciliation processes.
- Schedule and monitor ETL workflows using Databricks Workflows or orchestration tools such as Apache Airflow.
Preferred candidate profile
- Strong knowledge of ETL/ELT concepts, data integration, data extraction, transformation, and loading processes.
- Proficiency in Python programming, including functions, data structures, exception handling, and file processing.
- Hands-on experience with PySpark DataFrames, transformations, actions, joins, aggregations, and window functions.
- Experience with Databricks notebooks, jobs, clusters or compute, and Delta Lake.
- Robust SQL skills, including joins, CTEs, subqueries, window functions, aggregations, and query optimization.
- Understanding of Apache Spark architecture, lazy evaluation, partitioning, caching, shuffling, and performance tuning.
- Knowledge of data formats such as CSV, JSON, Avro, and Parquet, along with Delta Lake tables.
📌 Data Engineer- ETL (Hyderabad)
🏢 CGI
📍 Hyderabad