Responsibilities:

Data Pipeline Development

Design, develop, and maintain distributed data pipelines on AWS that ingest, process, and deliver data at scale. Implement both batch and event-driven data processing patterns using AWS-native services.

Event-Driven Architecture:

Build and support event-driven data solutions using asynchronous, decoupled architectures to enable near real-time processing and system scalability.

Technical Expertise

Provide hands-on engineering using Python and Apache Spark to process large datasets efficiently. Apply best practices in distributed systems, performance optimization, and fault tolerance.

Data Quality, Lineage & Orchestration:

Ensure end-to-end data quality, implement validation and monitoring checks, establish data lineage, and manage orchestration workflows to ensure reliable and auditable data movement.

API Development

Design and develop RESTful and event-driven APIs to expose data and data services to internal and external consumers.

Containers & Cloud-Native Development:





Build and deploy data services using Docker containers and manage workloads on Kubernetes following cloud-native and DevOps best practices.

Stakeholder Collaboration

Collaborate with data consumers, architects, platform teams, and business stakeholders to gather requirements, design scalable solutions, and deliver high-quality outcomes.

Operational Excellence

Contribute to monitoring, logging, alerting, and incident response for data platforms. Ensure systems meet reliability, performance, and security standards.

Requirements:
BS in Computer Science, Engineering, or equivalent practical experience
8+ years of experience in data engineering or distributed systems development
Solid hands-on experience with AWS cloud services
Advanced proficiency in Python
Hands-on experience with Apache Spark
Experience building event-driven data pipelines
Experience designing distributed data processing architectures
Robust knowledge of data quality, data lineage, and orchestration
Experience in API development
Hands-on experience with Docker and Kubernetes

📌 Senior Data Engineer Hyderabad (India)
🏢 LPL Financials
📍 India