Roles and Responsibilities :

- Design and implement complex data pipelines from disparate sources into our centralized data lake/warehouse.
- Utilize Apache Spark (via Dataproc) and Python/Scala for handling petabyte-scale transformations.
- Develop advanced SQL queries for data modeling, ensuring high performance in BigQuery.
- Create stream and batch processing solutions using Pub/Sub and Dataflow (Apache Beam).
- Manage workflow automation using Cloud Composer (Airflow).
- Architect solutions across Cloud Storage, Cloud SQL, and Spanner, selecting appropriate tools based on latency and consistency requirements.

Job Requirements :

- Proficiency in programming language Scala.
- Experience in tuning and debugging Spark jobs at scale.
- Hands-on experience with the full Google data suite (BigQuery, Dataflow, Dataproc, Pub/Sub).
- Ability to write and optimize complex queries for data modeling.
- Solid understanding of cloud-based technologies and data processing frameworks.
- Familiarity with Agile development methodologies and version control systems.

📌 Software Engineer (Pune)
🏢 CGI
📍 Pune