Seeking a Hadoop SME with 4+ years of overall experience and robust hands-on expertise in Big Data Engineering. The role will focus on designing, developing, and optimizing scalable data pipelines using Hadoop ecosystem technologies, covering batch, and streaming workloads, data integration, performance tuning, and collaboration with business and technical stakeholders.

Key Responsibilities

1. Design and develop scalable ETL and data pipelines using Hadoop ecosystem technologies.

2. Build, optimize, and troubleshoot large-scale data processing workloads using Apache Spark and Hive.

3. Develop and manage workflow orchestration using Apache Airflow.

4. Write complex SQL/HiveQL queries for data transformation, processing, and reporting.

5. Ingest structured and unstructured data from sources such as Kafka, Sqoop, APIs, and files.

6. Ensure data quality, lineage, governance, and reliability standards.

7. Perform performance tuning and optimization of Spark, Hive, and Hadoop workloads.





8. Troubleshoot production issues across distributed data-processing environments.

9. Create technical documentation and ensure adherence to coding and engineering best practices.

10. Provide technical guidance and collaborate with development, architecture, and business teams.

Mandatory Skills

- Hadoop ecosystem: HDFS, Hive

- Apache Spark

- Python and/or Scala

- Advanced SQL / HiveQL

- Apache Airflow

- ETL / Data Pipeline Development

- Data Modelling

- Linux / Unix and Shell Scripting

- Strong understanding of distributed computing and performance tuning

Good-to-Have Skills

- Apache Kafka

- Sqoop and API-based data ingestion

- Git/version control

- Agile delivery

- Data governance and lineage

- Production troubleshooting and optimization experience

📌 Hadoop SME (Bengaluru)
🏢 Talent21
📍 Bengaluru