Responsibilities
Design and build scalable big data processing applications.
Robust experience in Apache Spark, Scala, and distributed data processing; work on building high-performance data pipelines for analytics and data engineering use cases.
Develop and maintain data pipelines using Apache Spark (Scala).
Process large-scale datasets in distributed environments.
Implement batch and real-time data processing solutions.
Write effective and scalable code using Scala.
Work extensively with Spark Core, Spark SQL, and Spark Streaming.
Optimize Spark jobs for performance and resource utilization.
Ingest data from various sources: Databases (RDBMS, NoSQL), APIs, File systems (HDFS, S3).
Build ETL/ELT pipelines and data transformation workflows.
Tune Spark applications (partitioning, caching, memory tuning).
Optimize queries using Spark SQL.
Improve job execution time and reduce resource consumption.
Work with data engineers, data scientists, and business analysts.
Translate business requirements into scalable data solutions.
Participate in code reviews and architecture discussions.
Monitor data pipelines and Spark jobs.
Debug job failures and performance bottlenecks.
Ensure reliability and data quality in pipelines.
Technical and Skilled Requirements
Primary skills: Big Data - Data Processing - Spark; Functional Programming - Scala.
Preferred Skills
Functional Programming - Scala
Big Data - Data Processing - Spark