Senior Data Engineer - GCP, Big Query, PySpark & Kafka

Location

Hyderabad (Hybrid)

Experience

5 to 13 Years

Role Overview

We are looking for a highly skilled Data Engineer with strong expertise in building scalable, cloud-native data platforms on Google Cloud Platform (GCP). The ideal candidate should have extensive hands-on experience with Python, PySpark, BigQuery, Kafka, DBT, Terraform, Kubernetes, SQL, CI/CD, and ETL pipeline development. The candidate will be responsible for designing, developing, optimizing, and supporting large-scale data processing solutions that drive business insights and analytics.

Key Responsibilities

- Design, develop, and maintain scalable data pipelines and data processing frameworks on GCP.
- Build and optimize batch and real-time ETL/ELT pipelines using Python, PySpark, Kafka, and BigQuery.
- Develop high-performance data solutions for large-scale structured and unstructured datasets.
- Create and maintain data models using DBT and ensure adherence to data governance standards.
- Design streaming solutions using Kafka and event-driven architectures.
- Implement infrastructure as code (IaC) using Terraform for automated cloud provisioning.
- Deploy and manage containerized applications using Kubernetes.
- Build automated CI/CD pipelines for data platform deployments and releases.
- Optimize BigQuery performance, partitioning, clustering, and query tuning.
- Collaborate with data scientists, analysts, architects, and business stakeholders to build data products.
- Troubleshoot production issues and drive continuous improvements in platform reliability and performance.
- Ensure data quality, scalability, security, and operational excellence across the data ecosystem.

Must Have Skills

Data Engineering

- Python
- PySpark / Apache Spark




- Advanced SQL
- ETL / ELT Pipeline Development
- Data Warehousing Concepts
- Data Modeling

Google Cloud Platform (GCP)

- BigQuery
- Dataflow
- Pub/Sub
- Cloud Composer
- Cloud Storage (GCS)
- Cloud Functions (Positive to Have)

Streaming & Messaging

- Apache Kafka
- Event-Driven Architecture
- Real-Time Data Processing

DevOps & Cloud Engineering

- Terraform
- Kubernetes
- Docker
- CI/CD Pipelines
- Git/GitHub/GitLab

Data Transformation

- DBT (Data Build Tool)

Required Qualifications

- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.
- 5-13 years of experience in Data Engineering and Cloud-based Data Platforms.
- Strong hands-on expertise in GCP ecosystem and BigQuery.
- Experience building enterprise-grade data platforms and large-scale distributed data processing systems.
- Strong understanding of cloud-native architectures and modern data engineering practices.
- Excellent problem-solving and analytical skills.
- Strong communication and stakeholder management capabilities.

Preferred Skills

- Experience with Data Lake and Lakehouse architectures.
- Knowledge of Airflow/Cloud Composer.
- Exposure to BI/Analytics platforms such as Tableau, Looker, or Power BI.
- Experience in Financial Services, Banking, Insurance, or Enterprise Data Platforms.
- Exposure to Data Governance and Data Quality tools.

What We Are Looking For

- Strong hands-on coding experience in Python and PySpark.
- Deep expertise in BigQuery performance optimization.
- Proven experience with Kafka-based streaming solutions.
- Solid Terraform and Kubernetes implementation experience.
- Experience building robust CI/CD pipelines for data platforms.
- Ability to design end-to-end cloud-native data solutions on GCP.

📌 Gcp Data Engineer (Hyderabad)
🏢 Lloyds Technology Centre
📍 Hyderabad