Senior Data Engineer - Python | BigQuery | Databricks/Spark | Airflow

⚠️ This is NOT a typical Data Engineering position.

We're looking for a hands-on Data Engineer experienced in building scalable data pipelines using Python, BigQuery, and Databricks/Spark, with a strong understanding of workflow orchestration and production data processing.

If your experience is limited to writing SQL queries or maintaining basic ETL jobs, this role may not be the right fit. We need engineers who can build reliable pipelines, integrate data from multiple sources, orchestrate workflows, and troubleshoot data processing challenges at scale.

Job Title: Senior Data Engineer

Specialization: Data Engineering | Big Data | ETL/ELT | Data Pipeline Orchestration

Experience: 5+ Years

Location: BNG | MUM | PUN | HYD | GGN

Work Model: Hybrid

About the Role

We are looking for a Senior Data Engineer to build, manage, and optimize scalable data pipelines that power data-driven analytics and business insights.

You'll work across data ingestion, transformation, processing, and orchestration, using Python, BigQuery, Databricks/Spark, and Apache Airflow to develop reliable and efficient data workflows.

The role requires strong engineering fundamentals, hands-on pipeline development, data quality management, and the ability to optimize workflows for growing data volumes. You'll collaborate with technical teams and stakeholders to deliver accurate, reliable, and production-ready data solutions.

What You'll Do

- Design, develop, and maintain scalable ETL/ELT pipelines for large-scale data processing.
- Build data ingestion and transformation workflows using Python, BigQuery, and Databricks/Spark.




- Implement and manage workflow orchestration using Apache Airflow, including DAG development, task dependencies, scheduling, retries, and failure handling.
- Integrate data from multiple sources while ensuring consistency, accuracy, and reliability.
- Optimize data processing workflows to improve performance, scalability, and resource utilization.
- Develop and optimize SQL queries in BigQuery for large-scale data processing and transformation.
- Implement automated data validation and quality checks to maintain data integrity.
- Monitor pipeline execution, troubleshoot failures, investigate data discrepancies, and improve operational reliability.
- Manage pipeline dependencies and ensure timely, reliable execution of scheduled data workflows.
- Document data pipelines, transformation logic, dependencies, and engineering processes.
- Collaborate with engineering and analytics teams to translate data requirements into scalable technical solutions.

What We're Looking For

Mandatory Skills – Solid hands-on experience required:

- Python: Strong programming skills for data manipulation, transformation, and pipeline development.
- BigQuery: Hands-on experience with SQL, data processing, and query optimization.
- Databricks / Apache Spark:



Practical experience building and managing scalable big-data processing solutions.
- Apache Airflow: Experience developing and orchestrating workflows using DAGs, scheduling, dependencies, retries, and monitoring.
- ETL/ELT Pipeline Development: Experience designing, building, maintaining, and optimizing production data pipelines.
- SQL & Data Processing: Strong understanding of data transformation, integration, and working with large datasets.
- Data Quality & Troubleshooting: Ability to validate data, investigate pipeline failures, and maintain reliable data workflows.
- Experience: 5 years in Data Engineering or related roles involving large-scale data processing.

Good to Have

- Experience with Google Cloud Platform (GCP) and cloud-based data engineering.
- Exposure to Microsoft Azure and cloud data workflows.
- Experience implementing automated pipeline monitoring, logging, alerting, and recovery mechanisms.
- Understanding of data pipeline performance optimization and production support practices.

Why This Role Is Different

This isn't simply about writing SQL queries or running scheduled ETL jobs.

You'll build and orchestrate production-grade data pipelines that handle large datasets, integrate diverse data sources, and deliver reliable data at scale. The focus is on engineering robust workflows, improving performance, ensuring data quality, and making complex data processing reliable and repeatable.

Primary Skills (Mandatory)

Python | BigQuery | Databricks / Apache Spark | Apache Airflow | ETL/ELT | SQL | Data Pipeline Orchestration | Data Quality & Monitoring

📌 Senior Data Engineer (Bengaluru)
🏢 Publicis Global Delivery Pgd
📍 Bengaluru