Role description

Role: PySpark Application Support Lead

Experience: 8-12 Years

Location :Anywhere in India

Focus Production Support Troubleshooting Optimization Team Leadership

Role Summary

Leadership Skills and proven experience in leading technical teams handling escalations and conducting knowledgesharing sessions The Pyspark Application support is responsible for the monitoring and troubleshooting Pyspark Jobs on the Cloudera environment This role ensures high availability performance and security of ETL services while supporting developers analysts and business users across environments Prod and Nonprod

Technical Skills Primary

PySpark Application support

Handson experience with Pyspark jobs development ETL support

Monitor PySpark batch jobs streaming pipelines and scheduled workflows Airflow ADF Oozie etcChecking for failures and performance degradation

Troubleshoot job failures performance degradation schema mismatches data quality issues

Resolve errors related to Spark SQL DataFrame transformations cluster configuration and storage layers

Handle failures caused by upstreamdownstream systems Kafka Delta Lake Hive S3ADLS etc

Coordinating with development team for fixing the failures

Coordinating with QA team on data validation failures

Supporting in production release and change management process

Incident Problem Management

Root cause analysis for recurring issues

Performance tuning at a basic level cache queues agents

Familiarity with ITIL processes for AMS support

Tools

Ticketing systems ServiceNowJira

Monitoring tools

Version control basics GitSVN for migration scripts

Soft Skills

Ability to work with client stakeholders

Good communication with analytical and problemsolving skills

Ability to Effectively interact with stakeholders colleagues and other members of the team

Demonstrated ability to be organized and selfdirected





Strong interpersonal skills with the ability to work cross functionally

Responsibilities

Monitoring Maintenance

Proactively monitor Informatica environments for uptime and performance

Execute scheduled health checks and housekeeping tasks

Maintain logs and escalate anomalies promptly

Incident Management

Respond to incidents within SLA timelines

Perform initial triage resolve L2 issues and escalate L3 as needed

Document RCA and preventive measures

User Support

Assist users with access issues report failures and portal navigation

Manage user provisioning and security roles

Change Release Support

Support deployments and migrations between environments

Validate postdeployment checks and rollback plans

Compliance Documentation

Ensure adherence to security and audit requirements

Maintain SOPs runbooks and knowledge base for AMS operations

Production Support Incident Management Act as the primary point of contact for highpriority incidents debugging and resolving complex PySpark application issues in production

Technical Leadership Lead the efforts to design build and configure improvements to existing data pipelines and applications

Team Management Mentorship Guide mentor and manage a team of support specialistsdevelopers to achieve operational excellence and meet project deadlines

Performance Optimization Analyze existing PySpark applications and data pipelines for performance bottlenecks and implement optimization techniques eg Spark UI analysis tuning executors





Stakeholder Interaction Collaborate with crossfunctional teams developers Data Engineers DevOps to ensure seamless application functionality

Documentation Best Practices Create maintain and update technical documentation standard operating procedures SOPs and ensure code compliance with best practices

Automation Implement monitoring improvements and automate recurring manual tasks

Skills
Mandatory Skills : Apache Spark, Big Data Hadoop Ecosystem, SparkSQL

About LTM
LTM is an AI-centric global technology services company and the Business Creativity partner to the world’s largest and most disruptive enterprises. We bring human insights and intelligent systems together to help clients create greater value at the intersection of technology and domain expertise. Our capabilities span integrated operations, transformation, and business AI — enabling current ways of working, new productivity paradigms, and new roads to value. Together with over 87,000 employees across 40 countries and our global network of partners, LTM — a Larsen & Toubro company — owns business outcomes for our clients, helping them not just outperform the market, but to Outcreate it. Please also note that neither LTM nor any of its authorized recruitment agencies/partners charge any candidate registration fee or any other fees from talent (candidates) towards appearing for an interview or securing employment/internship. Candidates shall be solely responsible for verifying the credentials of any agency/consultant that claims to be working with LTM for recruitment. Please note that anyone who relies on the representations made by fraudulent employment agencies does so at their own risk, and LTM disclaims any liability in case of loss or damage suffered as a consequence of the same. Recruitment Fraud Alert - https://www.ltm.com/careers/recruitment-fraud-alert

📌 Senior Specialist - Data Engineering (Coimbatore)
🏢 LTM
📍 Coimbatore