Responsibilities

- ETL Development: Design and implement end-to-end ETL processes using AWS Glue (ETL jobs, Crawlers, Data Catalog) and AWS Lambda for event-driven data processing.
- Data Processing: Write high-performance PySpark scripts to transform large-scale datasets, ensuring optimal partitioning and resource utilization.
- Database Management: Build and optimize complex queries, schemas, and stored procedures in PostgreSQL to ensure data integrity and high performance.
- Pipeline Automation: Orchestrate workflows using AWS Step Functions or Airflow to automate data movement from various sources to our ODS and Data Lake.
- Performance Tuning: Monitor and optimize Glue job execution times, Lambda memory allocation, and PostgreSQL query execution plans.
- Data Quality: Implement automated data validation checks to ensure accuracy and consistency across the pipeline.
- Collaboration: Work closely with the Data Analytics team to translate business requirements into technical data models.

Requirements and skills

- Core Tech Stack: 3+ years of experience in Data Engineering with deep expertise in AWS Glue, AWS Lambda, and Amazon S3.
- PySpark Mastery:



Proven experience writing and debugging Spark/PySpark scripts for distributed data processing.
- PostgreSQL Fluency: Advanced SQL skills, including window functions, CTEs, and experience in performance tuning (indexing, vacuuming, and explain plans).
- Programming: Strong proficiency in Python for both data manipulation and serverless scripting.
- Cloud Architecture: Solid understanding of AWS IAM roles, VPC configurations for ETL jobs, and serverless best practices.
- Domain Knowledge: Prior experience in the Insurance or BFSI sector is a robust plus. Familiarity with insurance data models (Claims, Underwriting, Policy administration) will be highly valued.
- Education: Degree in Computer Science, Data Science, or a related field. AWS Certified Data Engineer Associate or Professional is a plus.

Preferred candidate profile

Min 2 years as AWS ETL Developer (preferred), PostgreSQL, Data Warehouse, Data Mart Design, DB development & maintenance

Immediate availability is preferred

📌 Aws Data Engineer (Chennai)
🏢 Galaxy Health Insurance
📍 Chennai