Dear All,

Greetings From Sparity..!!

Role: Technical Lead Data Engineering

Location: Offshore – India

Experience: 9 to 12 yrs

Tech Stack: AWS (Glue, Lambda, Step Functions, RDS, Aurora PostgreSQL, DynamoDB), PySpark, Python, SQL, Oracle, MongoDB, Streaming and Batch Data Integration, AI-Assisted Development

Working Knowledge: EKS, ECS, Docker, OpenShift, VPC, RDS, Security Groups, Terraform, GitHub, Jenkins, GitHub Actions, encryption, IAM

Role Overview

We are looking for a Lead Data Engineer to own the technical direction and delivery of data integration and data engineering work on AWS. Working alongside the Backend and Integration, UI/UX, and Automation teams, you will set engineering standards for batch and streaming pipelines, design for error handling and replay, and scale relational and NoSQL stores to multi-terabyte volumes. This role combines deep hands-on ownership with team leadership and acts as the primary technical escalation point for data pipeline design, performance, and reliability.

Key Responsibilities

- Lead the design, build, and delivery of ETL/ELT pipelines using AWS Glue and PySpark for batch and streaming data integration
- Design error handling, retry, and replay patterns so that pipelines recover cleanly and data is never silently lost
- Own data modeling and storage decisions across Aurora PostgreSQL, Oracle, MongoDB, DynamoDB, and RDS
- Lead the scaling of RDBMS and NoSQL systems to multi-terabyte volumes, including partitioning, indexing, and performance tuning
- Write and optimize complex SQL queries and stored procedures supporting reporting and downstream applications
- Perform data analysis and profiling to validate data quality, completeness, and consistency across pipelines
- Orchestrate pipelines using Lambda and Step Functions, and coordinate with the Backend and Integration team on events and APIs
- Define coding standards, design patterns, and best practices for data engineering across the team
- Own SDLC quality across design, configuration, testing, debugging,



implementation, and support, and work with the Automation Engineer on data validation tests
- Apply working knowledge of ECS, Terraform, GitHub, Jenkins, GitHub Actions, IAM, and encryption to deploy and secure pipelines
- Own production incident response and root-cause analysis for data pipeline issues, and implement long-term reliability improvements
- Mentor Senior Software Engineers, conduct code reviews, and document data flows and pipeline architecture

Required Qualifications
- Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field
- 9 to 12 years of experience in data engineering or ETL development, with demonstrated progression toward technical leadership
- Expert, hands-on experience with AWS Glue and PySpark (must haves)
- Strong knowledge of data integration patterns: streaming, batch, and error and replay
- Strong hands-on experience with Oracle, MongoDB, and DynamoDB (must haves), and with Aurora PostgreSQL and RDS
- Proven experience building and scaling RDBMS (PostgreSQL, Aurora PostgreSQL, Oracle) and NoSQL (MongoDB, DynamoDB) systems to multi-terabyte scale
- Hands-on knowledge of SQL and stored procedures, and strong data analysis techniques
- Strong proficiency in Python for scripting, automation, and Glue/PySpark development
- Experience across the full software development lifecycle: design, configuration, testing, debugging, implementation, and support of large-scale, business-centric, process-based applications
- Working awareness of EKS, ECS, Docker, OpenShift, VPC, RDS, Security Groups, Terraform, GitHub, Jenkins, GitHub Actions, encryption,



and Identity and Access Management (IAM); working knowledge of ECS and Terraform is a must have
- Strong debugging, performance-tuning, and production-support skills across complex pipelines
- Experience leading technical projects or guiding a team of data engineers
- Excellent written and verbal communication skills for cross-functional and stakeholder engagement

AI Knowledge & AI-Assisted Development (Required)
- Good understanding of AI/LLM models and how they can be applied to accelerate delivery, including prompt engineering
- Daily, practical use of AI development tools such as Devin AI, Cursor, and Claude Skills to accelerate development of Glue/PySpark scripts and SQL
- Able to critically review and validate AI-generated code, queries, and transformations for correctness, performance, and data integrity before deployment
- Defines and enforces team-level standards for responsible AI-assisted development, including validation rigor, security review of AI-generated code, and prompt hygiene
- Practical use of AI tools to assist with data profiling, anomaly detection, and technical documentation
- Understanding of secure and compliant AI tool usage, including never entering PHI, member data, client-specific credentials, or other sensitive information into prompts or external AI tools

Preferred Qualifications

- Experience in the US health insurance or payer domain: claims, eligibility, enrollment, provider, or member data, with HIPAA-aware system design and data handling practices
- Exposure to healthcare interoperability and data standards (HL7, FHIR, X12 EDI)
- Experience with additional AWS data services such as Athena, Redshift, Kinesis, or EMR
- Experience with contemporary orchestration tools (for example Apache Airflow)
- Relevant certifications: AWS Certified Data Engineer or Data Analytics Specialty

If interested, please apply here OR or share your CV to [email protected]

📌 Technical Lead Data Engineering_Sparity_Hyderabad (Hyderabad)
🏢 Sparity
📍 Hyderabad