Job Summary: We are seeking a Senior Data Engineer with 5+ years of experience in designing, developing, and maintaining scalable cloud-based data solutions. The ideal candidate should possess robust expertise in SQL, Python, PySpark, and Google Cloud Platform (GCP) services including BigQuery, Cloud Composer (Apache Airflow), Dataproc, and Pub/Sub. The role involves building enterprise-grade data pipelines, implementing scalable ETL/ELT solutions, and collaborating with Product, Engineering, and Business teams to deliver reliable, high-quality data products.
Responsibilities
Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.
Build and optimize cloud-native data solutions on Google Cloud Platform (GCP).
Develop optimized SQL queries and transformation logic in BigQuery.
Design and maintain orchestration workflows using Cloud Composer (Apache Airflow).
Develop, optimize, and troubleshoot Spark applications running on Dataproc.
Design event-driven data ingestion pipelines using Google Pub/Sub.
Ensure data quality through validation,



reconciliation, monitoring, and exception handling.
Develop reusable frameworks and utilities to improve engineering productivity.
Collaborate with Product Owners, Business SMEs, and Engineering teams to understand requirements and deliver scalable solutions.
Participate in code reviews and contribute to engineering best practices.
Create and maintain technical documentation for data pipelines and architecture.
Required Technical Skills
Python (Advanced)
SQL (Advanced)
PySpark
Google Cloud Platform (GCP)
BigQuery
Cloud Composer (Apache Airflow)
Dataproc
Pub/Sub
Cloud Storage
Data Engineering
ETL / ELT Development
Batch & Streaming Data Processing
Data Validation & Reconciliation
Pipeline Performance Optimization
Logging & Monitoring
Data Warehouse Concepts
Data Warehouse Architecture
Star & Snowflake Schema
Fact & Dimension Modeling
Slowly Changin

📌 Data Engineer Bengaluru
🏢 IBU Consulting
📍 Bengaluru