Senior Data Engineer - GCP, Big Query, PySpark & Kafka
Location
Hyderabad (Hybrid)
Experience
5 to 13 Years
Role Overview
We are looking for a highly skilled Data Engineer with robust expertise in building scalable, cloud-native data platforms on Google Cloud Platform (GCP). The ideal candidate should have extensive hands-on experience with Python, PySpark, BigQuery, Kafka, DBT, Terraform, Kubernetes, SQL, CI/CD, and ETL pipeline development. The candidate will be responsible for designing, developing, optimizing, and supporting large-scale data processing solutions that drive business insights and analytics.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and data processing frameworks on GCP.
- Build and optimize batch and real-time ETL/ELT pipelines using Python, PySpark, Kafka, and BigQuery.
- Develop high-performance data solutions for large-scale structured and unstructured datasets.
- Create and maintain data models using DBT and ensure adherence to data governance standards.
- Design streaming solutions using Kafka and event-driven architectures.




- Implement infrastructure as code (IaC) using Terraform for automated cloud provisioning.
- Deploy and manage containerized applications using Kubernetes.
- Build automated CI/CD pipelines for data platform deployments and releases.
- Optimize BigQuery performance, partitioning, clustering, and query tuning.
- Collaborate with data scientists, analysts, architects, and business stakeholders to build data products.
- Troubleshoot production issues and drive continuous improvements in platform reliability and performance.
- Ensure data quality, scalability, security, and operational excellence across the data ecosystem.

Must Have Skills
Data Engineering
- Python
- PySpark / Apache Spark
- Advanced SQL
- ETL / ELT Pipeline Development
- Data Warehousing Concepts
- Data Modeling

Google Cloud Platform (GCP)
- BigQuery
- Dataflow
- Pub/Sub
- Cloud Composer
- Cloud Storage (GCS)
- Cloud Func

📌 Gcp Data Engineer (Hyderabad)
🏢 Lloyds Technology Centre
📍 Hyderabad