Job Summary: We are seeking a Senior Data Engineer with 5+ years of experience in designing, developing, and maintaining scalable cloud-based data solutions. The ideal candidate should possess solid expertise in SQL, Python, PySpark, and Google Cloud Platform (GCP) services including BigQuery, Cloud Composer (Apache Airflow), Dataproc, and Pub/Sub. The role involves building enterprise-grade data pipelines, implementing scalable ETL/ELT solutions, and collaborating with Product, Engineering, and Business teams to deliver reliable, high-quality data products.

Responsibilities

- Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.
- Build and optimize cloud-native data solutions on Google Cloud Platform (GCP).
- Develop optimized SQL queries and transformation logic in BigQuery.
- Design and maintain orchestration workflows using Cloud Composer (Apache Airflow).
- Develop, optimize, and troubleshoot Spark applications running on Dataproc.
- Design event-driven data ingestion pipelines using Google Pub/Sub.
- Ensure data quality through validation, reconciliation, monitoring, and exception handling.
- Develop reusable frameworks and utilities to improve engineering productivity.
- Collaborate with Product Owners, Business SMEs, and Engineering teams to understand requirements and deliver scalable solutions.




- Participate in code reviews and contribute to engineering best practices.
- Create and maintain technical documentation for data pipelines and architecture.

Required Technical Skills

- Python (Advanced)
- SQL (Advanced)
- PySpark
- Google Cloud Platform (GCP)
- BigQuery
- Cloud Composer (Apache Airflow)
- Dataproc
- Pub/Sub
- Cloud Storage
- Data Engineering
- ETL / ELT Development
- Batch & Streaming Data Processing
- Data Validation & Reconciliation
- Pipeline Performance Optimization
- Logging & Monitoring
- Data Warehouse Concepts
- Data Warehouse Architecture
- Star & Snowflake Schema
- Fact & Dimension Modeling
- Slowly Changing Dimensions (SCD)
- Partitioning & Clustering
- Data Lake vs Data Warehouse concepts
- DevOps & Version Control
- Git / GitHub
- CI/CD Concepts
- Docker (Preferred)

Preferred Qualifications

- 5+ years of experience in Data Engineering.
- Strong hands-on experience with SQL, Python, and PySpark.
- Experience building production-grade data pipelines.
- Hands-on experience working with Google Cloud Platform (GCP).

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 data engineer (Bengaluru)
🏢 IBU Consulting
📍 Bengaluru