Senior Developer

Bengaluru, Karnataka

Job Summary

We are seeking a skilled GCP Data Engineer with 3-5 years of experience in designing, developing, and maintaining cloud-based data engineering solutions on Google Cloud Platform (GCP). The ideal candidate should have strong expertise in building end-to-end data pipelines from scratch using BigQuery, Airflow, Astronomer, Pub/Sub, Cloud Storage (GCS), and Python . The role requires hands-on experience in developing scalable, reliable, and high-performance data ingestion and transformation frameworks to support analytics and business intelligence initiatives.

Key Responsibilities

Design, develop, and maintain scalable data pipelines on Google Cloud Platform (GCP).Build end-to-end ETL/ELT solutions from scratch using GCP services and Python.Create, schedule, and manage complex Apache Airflow DAGs using Airflow and Astronomer.Develop robust data ingestion frameworks using Pub/Sub , batch processing, and real-time streaming approaches.Design and optimize data models, datasets, and transformations in BigQuery .Implement efficient data storage solutions using Google Cloud Storage (GCS) .Develop reusable and scalable Python-based data processing frameworks and utilities.Monitor pipeline performance, troubleshoot failures, and optimize processing efficiency.Ensure data quality through validations, reconciliations, and automated monitoring mechanisms.Collaborate with business stakeholders, analysts, and engineering teams to understand requirements and deliver scalable solutions.Implement security, governance, and best practices for data management on GCP.Manage deployment, orchestration,



and operational support of data pipelines in production environments.Participate in code reviews, architecture discussions, and continuous improvement initiatives.

Skill Requirements

Core GCP Skills 3-5 years of hands-on experience in Data Engineering and Cloud Data Platforms.Solid expertise in: Google BigQuery Cloud Storage (GCS) Pub/Sub Cloud Composer / Apache Airflow Astronomer GCP IAM and Security ConceptsExperience building cloud-native data solutions and pipelines from scratch on GCP. Data Engineering & Pipeline Development Hands-on experience in designing and developing ETL/ELT pipelines.Strong understanding of batch and streaming data processing architectures.Expertise in data ingestion, transformation, orchestration, and monitoring.Experience handling large-scale structured and semi-structured datasets.Good understanding of data warehousing concepts and dimensional modeling. Programming Skills Solid proficiency in Python for data engineering and pipeline development.Experience developing reusable frameworks, utilities, and automation scripts.Knowledge of error handling, logging, testing, and performance optimization techniques. Database & Analytics Strong SQL skills with experience in:Complex QueriesJoins and CTEsData Validation and ReconciliationQuery OptimizationPerformance TuningExperience supporting analytics and reporting use cases through optimized datasets. Preferred Skills Experience with CI/CD pipelines and DevOps practices.Exposure to DBT, Terraform, or Infrastructure-as-Code frameworks.Knowledge of containerization technologies such as Docker and Kubernetes.Experience working in Agile/Scrum teams.Exposure to BI tools such as Power BI, Tableau, or Looker.

Other Requirements

📌 Senior Developer (India)
🏢 HCLTech
📍 India