Job Description

Job title:Hadoop, Spark Engineer — Apache Spark / Scala / HadoopRole Summary:

We are seeking an experienced Big Data Engineer to design, build, and optimize large-scale batch and streaming data pipelines on the Hadoop ecosystem using Apache Spark and Scala. The role supports high-volume ingestion, transformation, and enrichment of clickstream, network, and location datasets, working closely with data architects, platform engineering, and downstream analytics teams. This is a hands-on engineering role with ownership of pipeline performance, reliability, and data quality in production.

Key Responsibilities

Design, develop, and maintain distributed data pipelines using Apache Spark (Core, SQL, Streaming) written in Scala.
Build ingestion and transformation workflows across the Hadoop ecosystem — HDFS, Hive, YARN, MapReduce — for structured and semi-structured data at TB–PB scale.
Tune and optimize Spark jobs: partitioning strategy, caching, broadcast joins, shuffle reduction, data skew handling, and executor/memory sizing.




Implement real-time and near-real-time ingestion using Apache NiFi and/or Kafka.
Embed data quality, reconciliation, and validation controls directly into pipelines.
Author and optimize HiveQL and Spark SQL for curated and consumption layers.
Automate orchestration and scheduling using Airflow, Oozie, or Control-M.
Participate in code reviews, CI/CD automation, unit and integration testing, and production support.
Troubleshoot job failures, SLA breaches, and performance regressions; drive root-cause analysis to permanent fixes.
Document data flows, lineage, transformation logic, and operational runbooks.

Required Qualifications:

4+ years of data engineering experience, with 2+ years hands-on Apache Spark development in Scala on production workloads.
Robust Scala fundamentals — functional programming constructs, collections API, case classes, pattern matching, implicit, and error handling.
Deep working knowledge of the Hadoop eco

📌 Senior Software Engineer (Chennai)
🏢 Infinite Computer Solutions
📍 Chennai