Job Summary: We are seeking a Senior Data Engineer with 5+ years of experience in designing, developing, and maintaining scalable cloud-based data solutions. The ideal candidate should possess robust expertise in SQL, Python, PySpark, and Google Cloud Platform (GCP) services including BigQuery, Cloud Composer (Apache Airflow), Dataproc, and Pub/Sub. The role involves building enterprise-grade data pipelines, implementing scalable ETL/ELT solutions, and collaborating with Product, Engineering, and Business teams to deliver reliable, high-quality data products.
Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.
- Build and optimize cloud-native data solutions on Google Cloud Platform (GCP).
- Develop optimized SQL queries and transformation logic in BigQuery.
- Design and maintain orchestration workflows using Cloud Composer (Apache Airflow).
- Develop, optimize, and troubleshoot Spark applications running on Dataproc.
- Design event-driven data ingestion pipelines using Google Pub/Sub.
- Ensure data quality through validation, reconciliation,



monitoring, and exception handling.
- Develop reusable frameworks and utilities to improve engineering productivity.
- Collaborate with Product Owners, Business SMEs, and Engineering teams to understand requirements and deliver scalable solutions.
- Participate in code reviews and contribute to engineering best practices.
- Create and maintain technical documentation for data pipelines and architecture.
Required Technical Skills
- Python (Advanced)
- SQL (Advanced)
- PySpark
- Google Cloud Platform (GCP)
- BigQuery
- Cloud Composer (Apache Airflow)
- Dataproc
- Pub/Sub
- Cloud Storage
- Data Engineering
- ETL / ELT Development
- Batch & Streaming Data Processing
- Data Validation & Reconciliation
- Pipeline Performance Optimization
- Logging & Monitoring
- Data Warehouse Concepts
- Data Warehouse Architecture
- Star & Snowflake Schema
- Fact & Dimension Modeling
- Slowly Changin

📌 data engineer (Bengaluru)
🏢 IBU Consulting
📍 Bengaluru