- Design and develop large-scale data processing pipelines using
- Build and maintain distributed applications utilizing Spark SQL, Spark Streaming, and DataFrames/Datasets. Optimize Spark jobs for performance, scalability, and resource utilization.
- Troubleshoot and resolve complex performance bottlenecks and production issues in a timely manner.