Required Technical Skills
Apache Flink
Solid hands-on experience in Apache Flink DataStream API.
Expertise in Apache Flink Table API & SQL API.
Experience designing and implementing: o Stateful stream processing o Event-time processing o Windowing operations o Checkpointing and fault tolerance o Watermarks and late event handling
Knowledge of Flink deployment and operations in distributed environments.
Apache Spark
Robust proficiency in: o Spark Core o Spark SQL o Structured Streaming
Experience developing high-volume ETL and data transformation pipelines.
Performance tuning and optimization of Spark jobs.
Knowledge of Spark execution architecture and resource management.
Responsibilities
Design, develop, and maintain scalable real-time data processing pipelines.
Build streaming applications using Apache Flink and Spark Structured Streaming.
Develop batch and near real-time ETL workflows.
Process and analyze large-scale datasets with low latency and high throughput.
Optimize data pipelines for performance, reliability, and scalability.
Collaborate with Data Engineers, Architects, and Business Stakeholders to deliver data solutions.
Implement monitoring, logging, and troubleshooting for data processing applications.
Ensure data quality, governance, and security standards are followed.