Role description

Job Title: Senior / Lead Big Data Engineer

Experience: 9 – 15 Years

Location: Bangalore (Hybrid)

Mode of interview: 2 Virtual+1 in-person

# of roles: 2

Notice period: Immediate to 30 Days joiners only

About the Role

We are looking for a seasoned Senior/Lead Data Engineer with 9 to 15 years of experience to architect, build, and optimize large-scale data platforms. In this role, you will bridge the gap between complex big data infrastructure, functional business domains, and enterprise AI initiatives. Beyond taking end-to-end ownership of batch data pipelines that process massive customer and behavioral datasets, you will play a key leadership role in building, mentoring, and scaling a high-performing data engineering team.

Key Responsibilities

Platform Optimization & Governance

Performance Tuning: Proactively troubleshoot and fine-tune complex Spark and Hive/HQL jobs to reduce latency, optimize memory usage, and manage cluster resource allocation (YARN).

Data Quality & Reliability: Establish automated data validation frameworks, lineage tracking, and strict SLA monitoring to guarantee data accuracy and platform reliability.

Platform Operations: Oversee on-prem big data platform stability, capacity planning, and workflow orchestration using Airflow.

Team Building & Technical Leadership

Build & Scale the Team: Partner with leadership to hire, onboard, and build a solid data engineering team from the ground up.

Mentorship & Guidance: Provide hands-on technical leadership, conduct architecture and code reviews, and mentor junior and mid-level engineers.

Engineering Standards: Establish and enforce best practices around Git version control, code quality, documentation, and Agile delivery methodologies.

Architecture & Pipeline Development

End-to-End ETL Ownership: Architect, develop, and maintain scalable, fault-tolerant batch ETL/ELT pipelines across on-premises Hadoop-based data platforms.

Large-Scale Data Processing:



Design high-throughput data ingestion and transformation workflows for massive customer and behavioral datasets.

Data Modeling: Lead dimensional and relational data modeling efforts to support enterprise analytics, reporting, and downstream AI solutions.

Functional Alignment & Cross-Team Collaboration

Business & Functional Translation: Translate complex functional business requirements into scalable technical data architectures, ensuring data assets directly solve business challenges.

Cross-Functional Partnership: Collaborate closely with analytics, product, and AI teams to understand data needs and deliver reliable, analytics-ready datasets.

Qualifications & Skills

Educational Qualification

B.E. or B.Tech in Computer Science, Information Technology, or a related engineering discipline.

Mandatory Qualifications

Experience: 9 to 15 years of progressive experience in Data Engineering, with deep hands-on expertise in Hadoop-based big data ecosystems (HDFS, YARN).

Leadership: Proven experience leading engineering initiatives and building or mentoring technical teams.

Big Data Stack: Expert-level proficiency in Apache Spark (Core, Spark SQL, PySpark) and Apache Hive / HQL.

Programming & Querying: Strong command of Python and advanced SQL, paired with solid experience in enterprise data modeling.

Pipeline Engineering: Proven track record of designing and maintaining production-grade batch ETL pipelines.

AI & Functional Acumen: Practical experience working on data initiatives supporting AI applications, combined with strong functional business understanding.

Version Control: Proficient in Git for collaborative software development.

Preferred Qualifications

Domain Experience: Prior experience working with e-commerce platforms and processing high-volume customer/transactional datasets is an added advantage.

Workflow Orchestration: Hands-on experience managing complex DAGs and schedules using Apache Airflow or Oozie.

Infrastructure & Tuning: Deep understanding of on-premise big data platform operations and advanced performance tuning for Hive and Spark workloads.

Methodology: Familiarity with Agile/Scrum development methodologies.

📌 Senior/Lead Big Data Engineer with Hadoop,Hive,Spark,SQL-Bangalore (Bengaluru)
🏢 Tredence
📍 Bengaluru