Required Technical Skills
- Apache Flink
- Solid hands-on experience in Apache Flink DataStream API.
- Expertise in Apache Flink Table API & SQL API.
- Experience designing and implementing: o Stateful stream processing o Event-time processing o Windowing operations o Checkpointing and fault tolerance o Watermarks and late event handling
- Knowledge of Flink deployment and operations in distributed environments.
- Apache Spark
- Strong proficiency in: o Spark Core o Spark SQL o Structured Streaming
- Experience developing high-volume ETL and data transformation pipelines.
- Performance tuning and optimization of Spark jobs.
- Knowledge of Spark execution architecture and resource management.
Responsibilities
- Design, develop, and maintain scalable real-time data processing pipelines.
- Build streaming applications using Apache Flink and Spark Structured Streaming.
- Develop batch and near real-time ETL workflows.
- Process and analyze large-scale datasets with low latency and high throughput.
- Optimize data pipelines for performance, reliability, and scalability.
- Collaborate with Data Engineers, Architects, and Business Stakeholders to deliver data solutions.
- Implement monitoring, logging, and troubleshooting for data processing applications.
- Ensure data quality, governance, and security standards are followed.