Role Summary: We are seeking a Senior Data Engineer with 5+ years of experience in designing, developing, and maintaining scalable cloud-based data solutions. The ideal candidate should possess strong expertise in SQL, Python, PySpark, and Google Cloud Platform (GCP) services including BigQuery, Cloud Composer (Apache Airflow), Dataproc, and Pub/Sub. The role involves building enterprise-grade data pipelines, implementing scalable ETL/ELT solutions, and collaborating with Product, Engineering, and Business teams to deliver reliable, high-quality data products.

Responsibilities

- Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.
- Build and optimize cloud-native data solutions on Google Cloud Platform (GCP).
- Develop optimized SQL queries and transformation logic in BigQuery.
- Design and maintain orchestration workflows using Cloud Composer (Apache Airflow).
- Develop, optimize, and troubleshoot Spark applications running on Dataproc.
- Design event-driven data ingestion pipelines using Google Pub/Sub.
- Ensure data quality through validation, reconciliation, monitoring, and exception handling.
- Develop reusable frameworks and utilities to improve engineering productivity.
- Collaborate with Product Owners, Business SMEs, and Engineering teams to understand requirements and deliver scalable solutions.
- Participate in code reviews and contribute to engineering best practices.
- Create and maintain technical documentation for data pipelines and architecture.

Requirements

Required Technical Skills

- Programming




- Python (Advanced)
- SQL (Advanced)
- PySpark
- Google Cloud Platform (GCP)
- BigQuery
- Cloud Composer (Apache Airflow)
- Dataproc
- Pub/Sub
- Cloud Storage
- Data Engineering
- ETL / ELT Development
- Batch Streaming Data Processing
- Data Validation Reconciliation
- Pipeline Performance Optimization
- Logging Monitoring
- Data Warehouse Concepts
- Data Warehouse Architecture
- Star Snowflake Schema
- Fact Dimension Modeling
- Slowly Changing Dimensions (SCD)
- Partitioning Clustering
- Data Lake vs Data Warehouse concepts
- DevOps Version Control
- Git / GitHub
- CI/CD Concepts
- Docker (Preferred)

Preferred Qualifications

- Experience working on enterprise-scale cloud migration initiatives.
- Experience building reusable data engineering frameworks.
- Knowledge of data governance and data quality practices.
- Exposure to AI-assisted development tools (e.g., GitHub Copilot) is an advantage.
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management skills.
- Experience 5+ years of experience in Data Engineering.
- Robust hands-on experience with SQL, Python, and PySpark.
- Experience building production-grade data pipelines.
- Hands-on experience working with Google Cloud Platform (GCP).

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 data engineer (Bengaluru)
🏢 Ibu
📍 Bengaluru