- Build batch and streaming data ingestion pipelines using Pub/Sub, Dataflow, Datastream (CDC), and Storage Transfer Service

- Load and ingest data into BigQuery via batch loads, the BigQuery Storage Write API and external/BigLake tables

- Manage Cloud Storage (GCS) landing zones, buckets, lifecycle policies and storage classes

- Develop ETL/ELT workflows using Dataflow (Apache Beam), Dataproc (Spark/Hadoop) and Dataprep

- Build SQL-based transformations using Dataform and BigQuery scheduled queries

- Implement medallion/layered architecture (raw → staging → curated/gold)

- Design and optimize BigQuery data warehouses with partitioning, clustering, materialized views and cost controls

- Orchestrate pipelines using Cloud Composer (Airflow) and Workflows

- Schedule and automate jobs with Cloud Scheduler and Cloud Functions

- Manage access and security using IAM, service accounts and Workload Identity Federation





- Implement data governance, lineage and quality using Knowledge Catalog

- Apply encryption (CMEK/CSEK, Cloud KMS), VPC Service Controls and DLP via Sensitive Data Protection

- Ensure compliance with PII masking and column/row-level security in BigQuery

- Enable analytics consumption through Looker, Looker Studio and BigQuery BI Engine

- Monitor pipelines and resources with Cloud Monitoring, Cloud Logging, and Cloud Trace

- Optimize query performance, slot usage, and overall cloud cost

- Implement CI/CD using Cloud Build, Artifact Registry, and Cloud Source Repositories

- Manage infrastructure-as-code using Terraform and Deployment Manager

- Maintain version control, automated testing and pipeline reliability with alerting and SLAs

📌 GCP Data Engineer (Hyderabad)
🏢 Paltech
📍 Hyderabad