Responsibilities:

- Design and implement complex data pipelines using Dataflow (Apache Beam) and Dataproc (Spark).
- Lead Cloud Composer (Airflow) orchestration design, optimizing DAG structure, dependencies, and retry strategies.
- Drive advanced Python and SQL implementations, including reusable frameworks and utility components.
- Build and standardize custom Dataflow / Beam templates and reusable pipeline patterns.
- Perform deep performance tuning for BigQuery (query optimization, partitioning strategies, cost optimization).
- Troubleshoot complex pipeline failures and performance bottlenecks across processing and orchestration layers.
- Review code and design artifacts produced by engineers; provide technical guidance and mentoring.
- Own technical quality for assigned modules, including non-functional aspects (performance, reliability, scalability).




- Act as the first point of escalation for technical issues before involving architects.
- Collaborate with architects to understand design intent and ensure correct implementation.

Mandatory Technical Skills:

- Advanced GCP Knowledge
- Strong understanding of IAM strategy, service isolation, and project-level design considerations.

- Programming & Querying

- Advanced proficiency in Python and complex GoogleSQL.

- Orchestration

- Robust hands-on expertise with Cloud Composer / Airflow, including best practices and optimization.

- Data Processing

- Deep hands-on experience with Dataflow and Apache Beam.
- Strong working knowledge of Dataproc / Spark internals.

- Analytics Platform

- Advanced experience with BigQuery, including performance and cost optimization techniques.

📌 Gcp Data Engineer (Hyderabad)
🏢 Elabs Infotech
📍 Hyderabad