Senior Apache Spark Technical Lead

Experience: Not Available to Not Available years

Location: Noida, India

Skills: Apache Spark, Scala, Python, Scuba Analytics, SQL, Python, Shell, CI/CD, Git, ETL, ELT, Data Quality, Grafana, Prometheus, CloudWatch, Kibana, ITSM, Incident Management, Problem Management, Change Management, Security, Compliance

Job Summary
We’re looking for a hands-on Scuba Developer cum Support Consultant to design, build, and support solutions on the Scuba platform. The role blends new feature development, data/modeling work, and support, ensuring platform stability, performance, and ongoing enhancements for business users and engineering stakeholders. To collaborate with product managers, platform engineers, data engineers, QA, and business SMEs, contributing across the SDLC—from requirements and design through deployment, monitoring, and incident/problem management.

Key Responsibilities
Development
• handling design and development, including data ingestion, quality, integration, analysis, modeling, and ETL
• Design, develop, and optimize Scuba dashboards/queries/models/pipelines (customize to the exact components your Scuba platform supports).
• Implement data integrations (batch/streaming), transformations, and enrichment aligned with data quality, lineage, and governance standards.
• Build reusable components, utilities, and automations (e.g., query templates, data loaders, CI/CD jobs).
• Participate in solution design reviews, code reviews, and performance tuning; ensure secure and scalable implementations.
• Author technical design documents, runbooks,



and user guides; contribute to internal knowledge base.

Support
• Provide support and solution for incidents, service requests, and problem records across Scuba environments (Dev/Test/Prod).
• Troubleshoot query performance, data freshness, access/permissions, and platform reliability issues; drive root cause analysis (RCA) and permanent fixes.
• Manage releases and change requests following ITSM and Change Advisory Board (CAB) processes; support UAT and production cutovers.
• Monitor SLAs/SLOs, capacity, and costs; implement alerts, health checks, and observability dashboards.
• Partner with stakeholders to prioritize backlog, communicate status/ETAs, and escalate risks proactively.

Skill Requirements
• Strong proficiency in apache spark, scala, and python
• Hands-on experience with Scuba (replace with: Scuba Analytics / your internal Scuba platform) for building dashboards, complex queries, and data models.
• Robust SQL (window functions, CTEs, performance tuning) and comfort with large datasets.
• Scripting for automation (e.g., Python or Shell) and basic CI/CD (Git, pipelines).
• Understanding of data pipelines/ETL/ELT, schema design, and data quality patterns.
• Familiarity with monitoring & logging (e.g., Grafana/Prometheus/CloudWatch, Kibana) and alerting.
• Solid grasp of ITSM (Incident/Problem/Change), SLAs, and on-call best practices.
• Security & compliance awareness (access controls/roles, PII handling, audit readiness).

Other Requirements
• Relevant certifications in apache spark, scala, or python are a plus

📌 Senior Apache Spark Technical Lead (Noida)
🏢 HCL Technologies
📍 Noida