Key Responsibilities

· Design, build, and optimize end-to-end data pipelines for batch and real-time data processing.

· Integrate data from various structured and unstructured data sources.

· Develop scalable and reusable ETL/ELT workflows following best practices.

· Work closely with Data Engineers, Architects, and Product/Business teams to understand data requirements.

· Ensure data accuracy, quality, governance, and lineage within pipelines.

· Monitor pipeline performance and proactively implement improvements.

· Automate workflows, scheduling, and monitoring using orchestration tools.

· Create and maintain documentation for data flows, data models, and pipeline configurations.

· Troubleshoot and resolve issues related to data availability, performance, or quality.

---

Required Skills & Experience

· 3–8 years of experience as Data Engineer / Data Pipeline Engineer.

· Strong hands-on expertise in:

o ETL/ELT frameworks

o Data pipeline development

o Distributed data processing (Spark/Databricks)

· Experience with one or more cloud platforms:

o GCP, AWS, Azure





· Solid programming knowledge in:

o Python (preferred)

o SQL (advanced level)

· Experience with cloud-native data tools such as:

o AWS Glue, Step Functions

o Azure Data Factory

o GCP Dataflow/Composer/Dataproc

· Experience with data warehousing technologies such as:

o Snowflake, BigQuery, Redshift, Synapse

· Familiar with orchestration & workflow management tools:

o Airflow, Prefect, Luigi, Composer, AWS MWAA

· Strong understanding of data modelling, data quality frameworks, and data governance principles.

· Knowledge of CI/CD pipelines, Git, DevOps practices is a plus.

---

Preferred Skills

· Experience in Databricks (highly preferred if using Spark).

· Knowledge of Kafka, Pub/Sub, Kinesis, or other streaming services.

· Exposure to DBT for data transformations.

· Familiarity with containerization tools (Docker/Kubernetes).

· Background in building data lakes/lakehouse arc

📌 GCP Data Engineer (Bengaluru)
🏢 TalentOla
📍 Bengaluru