Key Responsibilities

· Design, build, and optimize end-to-end data pipelines for batch and real-time data processing.

· Integrate data from various structured and unstructured data sources.

· Develop scalable and reusable ETL/ELT workflows following best practices.

· Work closely with Data Engineers, Architects, and Product/Business teams to understand data requirements.

· Ensure data accuracy, quality, governance, and lineage within pipelines.

· Monitor pipeline performance and proactively implement improvements.

· Automate workflows, scheduling, and monitoring using orchestration tools.

· Create and maintain documentation for data flows, data models, and pipeline configurations.

· Troubleshoot and resolve issues related to data availability, performance, or quality.

---

Required Skills & Experience

· 3–8 years of experience as Data Engineer / Data Pipeline Engineer.

· Strong hands-on expertise in:

o ETL/ELT frameworks o Data pipeline development o Distributed data processing (Spark/Databricks)

· Experience with one or more cloud platforms:

o GCP, AWS, Azure

· Strong programming knowledge in:

o Python (preferred)

o SQL (advanced level)

· Experience with cloud-native data tools such as:

o AWS Glue,



Step Functions o Azure Data Factory o GCP Dataflow/Composer/Dataproc

· Experience with data warehousing technologies such as:

o Snowflake, BigQuery, Redshift, Synapse

· Familiar with orchestration & workflow management tools:

o Airflow, Prefect, Luigi, Composer, AWS MWAA

· Strong understanding of data modelling, data quality frameworks, and data governance principles.

· Knowledge of CI/CD pipelines, Git, DevOps practices is a plus.

---

Preferred Skills

· Experience in Databricks (highly preferred if using Spark).

· Knowledge of Kafka, Pub/Sub, Kinesis, or other streaming services.

· Exposure to DBT for data transformations.

· Familiarity with containerization tools (Docker/Kubernetes).

· Background in building data lakes/lakehouse architectures.

---

Education

· Bachelor's degree in Computer Science, Engineering, IT, or related field.

· Certifications in cloud technologies (AWS/GCP/Azure) are an added advantage.

---

Soft Skills

· Solid analytical and problem-solving abilities.

· Excellent communication and stakeholder management skills.

· Ability to work in an agile, fast-paced, and collaborative environment.

📌 GCP Data Engineer (India)
🏢 TalentOla
📍 India