18 Nov
|
Talent21
|
Chennai
Job description
Seeking a Hadoop SME with 4+ years of overall experience and solid hands-on expertise in Big Data Engineering. The role will focus on designing, developing, and optimizing scalable data pipelines using Hadoop ecosystem technologies, covering batch, and streaming workloads, data integration, performance tuning, and collaboration with business and technical stakeholders.
Key Responsibilities
Design and develop scalable ETL and data pipelines using Hadoop ecosystem technologies.
Build, optimize, and troubleshoot large-scale data processing workloads using Apache Spark and Hive.
Develop and manage workflow orchestration using Apache Airflow.
Write complex SQL/HiveQL queries for data transformation, processing, and reporting.
Ingest structured and unstructured data from sources such as Kafka, Sqoop, APIs, and files.
Ensure data quality, lineage, governance, and reliability standards.
Perform performance tuning and optimization of Spark, Hive,
and Hadoop workloads.
Troubleshoot production issues across distributed data-processing settings.
Create technical documentation and ensure adherence to coding and engineering best practices.
Provide technical guidance and collaborate with development, architecture, and business teams.
Mandatory Skills
Hadoop ecosystem: HDFS, Hive
Apache Spark
Python and/or Scala
Advanced SQL / HiveQL
Apache Airflow
ETL / Data Pipeline Development
Data Modelling
Linux / Unix and Shell Scripting
Robust understanding of distributed computing and performance tuning
Good-to-Have Skills
Apache Kafka
Sqoop and API-based data ingestion
Git/version control
Agile delivery
Data governance and lineage
Production troubleshooting and optimization experience
📌 Hadoop Sme Chennai
🏢 Talent21
📍 Chennai