- Experience in designing scalable data pipelines on GCP with expertise in Dataflow, Pub/Sub, BigQuery, Python, and SQL
- Develop and maintain batch and real-time data processing solutions, optimize data warehouse performance, implement ETL/ELT workflows, and manage workflow orchestration using Cloud Composer/Airflow
Role & responsibilities
- Design, develop, and maintain scalable data pipelines on GCP.
- Build batch and real-time data processing solutions using Dataflow and Pub/Sub.
- Develop and optimize data warehouse solutions using BigQuery.
- Build ETL/ELT workflows for structured and semi-structured data.
- Develop data processing scripts and applications using Python and SQL.
- Work with Cloud Storage for ingestion and data-lake use cases.
- Perform data transformation, validation, cleansing, and quality checks.
- Optimize BigQuery queries, partitioning, clustering, and overall pipeline performance.
- Implement workflow orchestration using tools such as Cloud Composer / Apache Airflow.
- Implement monitoring, logging, error handling, and production support processes.
- Collaborate with data architects, analysts, application teams, and DevOps teams.
- Follow security and access-control practices using IAM and service accounts.