Job Summary

We are looking for an experienced Data Engineer with robust expertise in distributed data processing, real-time streaming, and data modeling. The ideal candidate should have hands-on experience with Apache Spark, Apache Flink, Kafka, and SQL to design, develop, and maintain scalable batch and streaming data pipelines.

Key Responsibilities

- Design, develop, and maintain scalable data pipelines using Apache Spark, PySpark, and Apache Flink.
- Build real-time data ingestion and streaming solutions using Apache Kafka.
- Develop batch and stream-processing applications for large-scale data processing.
- Design, implement, and maintain data models to support analytical and business reporting requirements.
- Perform data transformation, cleansing, aggregation, and validation across multiple data sources.
- Optimize Spark jobs, streaming applications, SQL queries, and data processing workflows for performance.
- Work with structured and semi-structured data in distributed computing environments.
- Implement data quality checks, error handling, monitoring, and pipeline reliability mechanisms.
- Collaborate with data architects, data analysts, data scientists, and business teams to understand data requirements.
- Follow best practices for data governance, security, scalability, and documentation.

Required Skills

- Strong hands-on experience with Apache Spark and PySpark.




- Experience with Apache Flink for real-time stream processing.
- Good knowledge of Apache Kafka, including topics, partitions, producers, consumers, and event streaming.
- Strong SQL skills and experience working with large datasets.
- Experience in data modeling, including conceptual, logical, and physical data models.
- Knowledge of ETL/ELT processes and data pipeline development.
- Understanding of distributed computing, parallel processing, and data processing optimization.
- Experience with data quality, data validation, and troubleshooting pipeline issues.

Good to Have

- Experience with cloud data platforms such as AWS, Azure, or GCP.
- Knowledge of data lakes, data warehouses, and lakehouse architectures.
- Familiarity with Delta Lake, Apache Iceberg, or Apache Hudi.
- Experience with workflow orchestration tools such as Airflow.
- Knowledge of schema management and formats such as Avro, Parquet, and JSON.
- Experience with data governance, metadata management, and data cataloguing.

Candidate Profile

The ideal candidate should have practical experience building scalable batch and real-time data processing solutions using Spark, Flink, and Kafka, along with strong data modeling and SQL capabilities.

Key Skills for Sourcing: Data Engineer, Apache Spark, PySpark, Apache Flink, Kafka, Data Modeling, SQL, ETL/ELT, Real-Time Streaming, Batch Processing.

📌 Data Engineer (Pune)
🏢 Nustar Technologies
📍 Pune