17 Nov
|
Talent21
|
Bengaluru
17 Nov
Talent21
Bengaluru
– Big Data SME / Architect
Position: Big Data SME / Architect
Experience: 4–10 Years
Location: Chennai (Sholinganallur) / Bangalore (RMZ Ecoworld)
Work Mode: Hybrid
Budget: Up to 40 LPA
Employment Type: Full-Time
Job Overview
Tech Mahindra is looking for an experienced Big Data SME / Architect with 4–10 years of overall experience and strong hands-on expertise in Big Data Engineering.
The ideal candidate should have extensive experience in designing, developing, and optimising scalable data pipelines using Hadoop ecosystem technologies, including Apache Hive, Spark, PySpark, and Airflow. The role involves handling large-scale data processing, ETL development, workflow orchestration, performance optimisation, and providing technical leadership.
Key Responsibilities
1. Design and develop scalable ETL processes and data pipelines using Hadoop ecosystem technologies.
2. Build, optimise, and troubleshoot large-scale data processing workloads using Apache Spark, PySpark, and Hive.
3. Design, develop, and manage workflow orchestration using Apache Airflow.
4. Write complex SQL and HiveQL queries for data transformation and processing.
5. Handle structured and unstructured data ingestion from Kafka, Sqoop, APIs, and other sources.
6. Ensure data quality, data governance, lineage, and pipeline reliability.
7. Perform performance tuning and optimisation of Spark, Hive, and Hadoop workloads.
8. Troubleshoot complex issues in distributed data-processing environments.
9.
Develop technical documentation and maintain engineering best practices.
10. Provide technical guidance and collaborate with architecture, development, and business teams.
Mandatory Skills
- Hadoop Ecosystem – HDFS, Hive
- Apache Spark and PySpark
- Apache Airflow
- Python and/or Scala
- Advanced SQL / HiveQL
- ETL and Data Pipeline Development
- Data Modelling
- Linux / Unix and Shell Scripting
- Distributed Computing
- Performance Tuning and Optimisation
Good-to-Have Skills
- Apache Kafka
- Sqoop and API-based Data Ingestion
- Git / Version Control
- Agile Methodology
- Data Governance and Data Lineage
- Production Support and Troubleshooting
- Large-scale Data Processing
Experience & Qualification
- 4–10 years of overall IT experience.
- Strong hands-on experience in Big Data Engineering.
- Expertise in Hive, Spark, PySpark, and Airflow.
- Experience designing and optimising enterprise-level data pipelines.
- Strong problem-solving and technical leadership capabilities.
Preferred Candidate Profile
Candidates should have robust hands-on technical expertise in Big Data technologies along with solution design and architecture capabilities. The ideal candidate should be capable of handling complex data engineering challenges, optimising distributed workloads, and guiding technical teams.
Note: Candidates with strong hands-on experience in Hive, Spark, PySpark, and Airflow will be preferred.
📌 Bigdata and Hadoop Developer (Bengaluru)
🏢 Talent21
📍 Bengaluru