Job Requirements

- - Design and develop robust data ingestion and ETL pipelines for batch and workflow-based processing requirements.

- Build, optimize, and maintain data processing solutions using Apache Spark, PySpark, and Apache NiFi.

- Develop, schedule, monitor, and manage workflows in Apache Airflow to ensure reliable orchestration.

- Work on the Cloudera Data Engineering platform and the broader Hadoop ecosystem, including HDFS and Hive.

- Write efficient Hive SQL and Python code for data transformation, validation, and operational automation.

- Support, troubleshoot, and optimize production data pipelines for performance, reliability, and scalability.

- Collaborate with data architects, analysts, platform teams, and business stakeholders to understand requirements and deliver data solutions.

- Follow engineering best practices for code quality, version control, documentation, deployment,



and monitoring.

Work Experience

- 8+ years of relevant experience in data engineering, big data platforms, and production pipeline delivery.

- Strong hands-on experience in designing data pipelines using NiFi, Spark/PySpark, Hive SQL, and Python.

- Valuable understanding of Hadoop concepts, HDFS storage patterns, data partitioning, and distributed processing.

- Experience managing Airflow DAGs, dependencies, retries, monitoring, and failure handling.

- Ability to identify pipeline bottlenecks and perform performance tuning across ingestion, transformation, and orchestration layers.

Strong problem-solving mindset with the ability to support critical production workloads.

📌 Data Engineering Specialist (Pune)
🏢 Quest Global
📍 Pune