Role & responsibilities

- Develop data transformation logic using Python, SQL and PySpark where applicable.
- Design, develop and maintain scalable and reliable data pipelines using Apache Airflow and AWS services.
- Build and optimize ETL/ELT workflows to ingest, transform and process data from various source systems.
- Develop and manage Airflow DAGs, task dependencies, scheduling, monitoring and error handling.
- Implement data ingestion solutions using AWS services such as S3, Glue, Lambda, Step Functions, EventBridge, Athena and RDS/Aurora.
- Monitor data pipeline performance and troubleshoot workflow failures, data quality issues and operational incidents.
- Collaborate with architects, business analysts and application teams to understand data requirements and deliver scalable solutions.
- Support production deployments, incident management, root cause analysis and performance tuning activities.
- Implement data validation, reconciliation and quality checks to ensure accurate and reliable data delivery.
- Participate in cloud modernization and migration initiatives to AWS-based platforms.
- Contribute to CI/CD implementation, automation, documentation and engineering best practices.
- Develop and maintain operational dashboards, monitoring and alerting solutions for pipeline observability.
- Participate in Agile ceremonies, sprint planning and technical design discussions.
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- 4-6 years of experience in Data Engineering,



ETL Development, or Data Integration.
- Robust proficiency in Python programming.
- Strong hands-on experience with Apache Airflow for workflow orchestration and scheduling.
- Experience with AWS cloud services, including:
- Amazon S3, Glue, Lambda, Step Functions, Athena, EventBridge, RDS/Aurora PostgreSQL, CloudWatch
- Strong SQL skills with experience in relational databases such as PostgreSQL, Oracle, or SQL Server.
- Experience developing and supporting batch and event-driven data pipelines.
- Understanding of data modeling, data warehousing and ETL/ELT concepts.
- Experience with source control systems such as Git.
- Knowledge of monitoring, logging and troubleshooting distributed data workflows.
- Strong analytical, problem-solving and debugging skills.
- Excellent verbal and written communication skills.

Preferred candidate profile

- Experience with PySpark and distributed data processing frameworks.
- Knowledge of AWS EMR, Redshift, or Snowflake.
- Experience working with Docker and containerized applications.
- Exposure to Kubernetes or Amazon EKS.
- Familiarity with REST API integration and event-driven architectures.
- Experience implementing CI/CD pipelines using GitHub Actions, GitLab CI, or AWS CodePipeline.
- Understanding of data governance, metadata management and data lineage concepts.
- Experience working in Agile/Scrum environments.
- Exposure to infrastructure-as-code tools such as Terraform or AWS CloudFormation.

📌 AWS & Airflow Developer (Hyderabad)
🏢 LPL Financials
📍 Hyderabad