Job Description

Job Description

Job title:Hadoop, Spark Engineer — Apache Spark / Scala / HadoopRole Summary:

We are seeking an experienced Big Data Engineer to design, build, and optimize large-scale batch and streaming data pipelines on the Hadoop ecosystem using Apache Spark and Scala. The role supports high-volume ingestion, transformation, and enrichment of clickstream, network, and location datasets, working closely with data architects, platform engineering, and downstream analytics teams. This is a hands-on engineering role with ownership of pipeline performance, reliability, and data quality in production.

Key Responsibilities
- Design, develop, and maintain distributed data pipelines using Apache Spark (Core, SQL, Streaming) written in Scala.
- Build ingestion and transformation workflows across the Hadoop ecosystem — HDFS, Hive, YARN, MapReduce — for structured and semi-structured data at TB–PB scale.
- Tune and optimize Spark jobs: partitioning strategy, caching, broadcast joins, shuffle reduction, data skew handling, and executor/memory sizing.




- Implement real-time and near-real-time ingestion using Apache NiFi and/or Kafka.
- Embed data quality, reconciliation, and validation controls directly into pipelines.
- Author and optimize HiveQL and Spark SQL for curated and consumption layers.
- Automate orchestration and scheduling using Airflow, Oozie, or Control-M.
- Participate in code reviews, CI/CD automation, unit and integration testing, and production support.
- Troubleshoot job failures, SLA breaches, and performance regressions; drive root-cause analysis to permanent fixes.
- Document data flows, lineage, transformation logic, and operational runbooks.

Required Qualifications:
- 4+ years of data engineering experience, with 2+ years hands-on Apache Spark development in Scala on production workloads.
- Solid Scala fundamentals — functional programming constructs, collections API, case classes, pattern matching, implicit, and error handling.
-

📌 Senior Software Engineer (Chennai)
🏢 Infinite Computer Solutions
📍 Chennai