Role & responsibilities

- Build batch and streaming data ingestion pipelines using Pub/Sub, Dataflow, Datastream (CDC), and Storage Transfer Service
- Load and ingest data into BigQuery via batch loads, the BigQuery Storage Write API and external/BigLake tables
- Manage Cloud Storage (GCS) landing zones, buckets, lifecycle policies and storage classes
- Develop ETL/ELT workflows using Dataflow (Apache Beam), Dataproc (Spark/Hadoop) and Dataprep
- Build SQL-based transformations using Dataform and BigQuery scheduled queries
- Implement medallion/layered architecture (raw staging curated/gold)
- Design and optimize BigQuery data warehouses with partitioning, clustering, materialized views and cost controls
- Orchestrate pipelines using Cloud Composer (Airflow) and Workflows
- Schedule and automate jobs with Cloud Scheduler and Cloud Functions
- Manage access and security using IAM, service accounts and Workload Identity Federation




- Implement data governance, lineage and quality using Knowledge Catalog
- Apply encryption (CMEK/CSEK, Cloud KMS), VPC Service Controls and DLP via Sensitive Data Protection
- Ensure compliance with PII masking and column/row-level security in BigQuery
- Enable analytics consumption through Looker, Looker Studio and BigQuery BI Engine
- Monitor pipelines and resources with Cloud Monitoring, Cloud Logging, and Cloud Trace
- Optimize query performance, slot usage, and overall cloud cost
- Implement CI/CD using Cloud Build, Artifact Registry, and Cloud Source Repositories
- Manage infrastructure-as-code using Terraform and Deployment Manager
- Maintain version control, automated testing and pipeline reliability with alerting and SLAs

📌 GCP Data Engineer (Hyderabad)
🏢 Paltech Consulting
📍 Hyderabad