11 Sep
|
Particleblack
|
India
11 Sep
Particleblack
India
● Google Cloud Platform (GCP) Tools:(Mandatory) - BigQuery, Cloud Storage, Dataflow, Cloud Functions, Pub/Sub, Cloud Run, Cloud Composer (Airflow), Cloud Spanner, Bigtable.
● Container Orchestration - Kubernetes (preferred on GKE) and Helm for managing and deploying containerized applications.
● CI/CD and Automation- Jenkins for building CI/CD pipelines to automate deployment and testing of data pipelines.
●Develop and Optimize Pipelines: Write efficient Python and SQL scripts to build data pipelines and ETL/ELT processes. Continuously monitor and optimize data workflows for performance and cost-effectiveness.
● Data Integration and Orchestration: Design workflows to integrate data from various sources using GCP services, and orchestrate complex tasks with Cloud Composer (Apache Airflow)
● Programming Languages- Proficient in Python for data processing and automation, SQL for querying and data manipulation. Experience with Java is a plus.
● DevOps Tools - Familiarity with Terraform or Deployment Manager for Infrastructure as Code (IaC) to manage GCP resources.
● Monitoring and Logging -Experience with Cloud Monitoring, Datadog, or other monitoring solutions to track pipeline performance and ensure operational efficiency.
● Data Engineering Skills Expertise in ETL/ELT pipelines, data modeling, and data integration across large datasets.
● Strong understanding of data warehousing and real-time data processing workflows.
● Strong communication skills to work effectively with cross-functional teams and mentor junior developers. Proven ability to lead in an Agile workplace.
● 3+ years of experience as a data engineer, with hands-on experience in Kubernetes, Helm, Python and Jenkins.
● Strong experience building and optimizing data pipelines and services in any cloud platform.
● Proficiency in Python and SQL. Familiarity with Java and Docker is a plus.
📌 Senior Data Engineer (India)
🏢 Particleblack
📍 India