Key Responsibilities

- Data Pipeline Development
- Design and implement robust, scalable batch and streaming data pipelines using Python and GCP services.
- Build and maintain data transformation workflows in Dataform (SQLX) integrated with BigQuery.
- Develop event-driven and real-time data processing solutions using Pub/Sub and Cloud Functions / Cloud Run.
- Manage large-scale structured and unstructured data stored in GCS, BigQuery, and Cloud Spanner.
- Python & Google SDK
- Write clean, well-structured Python code leveraging GCP client libraries (google-cloud-bigquery, google-cloud-storage, google-cloud-pubsub, google-cloud-spanner, etc.).
- Develop reusable Python modules, utilities, and SDKs for internal data platform tooling.
- Ensure all code meets quality standards through thorough unit testing with pytest and code reviews.
- Infrastructure & DevOps
- Provision and manage GCP infrastructure using Terraform, following infrastructure-as-code (IaC) best practices.
- Maintain CI/CD pipelines in GitLab for automated testing, linting, and deployment of data workloads.
- Understanding of monitoring, alerting, and observability for data pipelines and GCP resources.
- Manage containerised workloads on Cloud Run, including Dockerfiles and container registries.

Skills

- 5+ years of experience in data engineering with at least 2 years on GCP.
- Strong Python programming skills clean code, OOP, packaging,



and performance optimization.
- Hands-on experience with BigQuery: query optimization, partitioning, clustering, and data modelling.
- Experience with Pub/Sub for event-driven architectures and streaming ingestion.
- Proficiency in Cloud Run and/or Cloud Functions for serverless and containerized workloads.
- Experience writing unit tests with pytest mocking, fixtures, coverage reporting.
- Hands-on Terraform experience for GCP resource provisioning (projects, IAM, networking, data services).
- Experience with GitLab CI/CD pipeline authoring, environment management, secrets handling.
- Solid understanding of GCS for data lake patterns file formats (Parquet, Avro, JSON), lifecycle policies.

Nice to Have

- Experience with Dataform for SQL-based transformation pipelines in BigQuery.
- Familiarity with Cloud Spanner for globally distributed relational workloads.
- Knowledge of data mesh or data lakehouse architecture patterns.
- Knowledge/Experience with Apache Beam / Dataflow for large-scale streaming or batch processing.
- GCP Qualified Data Engineer or Cloud Architect certification.
- Familiarity with Apache Airflow / Cloud Composer, or similar orchestration tools.

Additional Information:

- The candidate should have minimum 3 years of experience in Google Cloud Platform Architecture.

📌 Custom Software Engineer (Pune)
🏢 hackajob
📍 Pune