- Design and build scalable big data processing applications.
- Strong experience in Apache Spark, Scala, and distributed data processing; work on building high-performance data pipelines for analytics and data engineering use cases.
- Develop and maintain data pipelines using Apache Spark (Scala).
- Process large-scale datasets in distributed environments.
- Implement batch and real-time data processing solutions.
- Write effective and scalable code using Scala.
- Work extensively with Spark Core, Spark SQL, and Spark Streaming.
- Optimize Spark jobs for performance and resource utilization.
- Ingest data from various sources: Databases (RDBMS, NoSQL), APIs, File systems (HDFS, S3).
- Build ETL/ELT pipelines and data transformation workflows.
- Tune Spark applications (partitioning, caching, memory tuning).
- Optimize queries using Spark SQL.
- Improve job execution time and reduce resource consumption.
- Work with data engineers, data scientists, and business analysts.
- Translate business requirements into scalable data solutions.
- Participate in code reviews and architecture discussions.
- Monitor data pipelines and Spark jobs.
- Debug job failures and performance bottlenecks.
- Ensure reliability and data quality in pipelines.
Technical and Professional Requirements
- Primary skills: Big Data - Data Processing - Spark; Functional Programming - Scala.
Preferred Skills
- Functional Programming - Scala
- Big Data - Data Processing - Spark