Job Description
About Kagool
nWe are a fast-growing IT consultancy specializing in the transformation of complex Global enterprises that use SAP. We are looking for hard working individuals to help deliver for our global customer base. We embrace the opportunities of the future and work proactively to make good use of technology. As you can imagine, this means that we have a vibrant and diverse mix of skills and people making Kagool a great place to work.
n

nJob Summary
nWe are looking for an experienced Senior Data Engineer with 7+ years of experience in designing, developing, and supporting scalable data engineering solutions.
nThe ideal candidate must have solid hands-on experience with Apache Spark, PySpark, Kafka, Spark Structured Streaming, Python, SQL, and Microsoft Azure. The candidate should have practical experience implementing and managing Kafka-based streaming solutions on Azure and working with real-time data processing pipelines.
n

nAzure experience is mandatory for this role.
n

nKey Responsibilities
n
- n
- Design, develop,



and maintain scalable batch and real-time data pipelines using Apache Spark and PySpark.n
- Develop and support Kafka-based real-time streaming pipelines on Microsoft Azure.n
- Implement real-time data processing using Spark Structured Streaming.n
- Work with Kafka topics, partitions, consumer groups, offsets, retention, and message delivery mechanisms.n
- Implement Kafka producers and consumers for high-volume data ingestion and processing.n
- Integrate Apache Kafka with Azure data services and downstream data platforms.n
- Develop data transformation, cleansing, aggregation, and enrichment processes.n
- Implement checkpointing, fault tolerance, error handling, retry mechanisms, and recovery strategies for streaming applications.n
- Design and optimize Spark jobs for performance, scalability, and efficient resource utilization.n
- Work with Azure data services such as Azure Databricks, ADLS Gen2, Azure Eve

📌 Senior Data Engineer (Noida)
🏢 Kagool
📍 Noida