We are looking for an experienced Senior Data Engineer with strong expertise in designing, developing, monitoring, and optimizing scalable data pipelines. The ideal candidate should have hands-on experience with AWS, PySpark, AWS Glue, Airflow, S3/Iceberg, Redshift, and PostgreSQL.
Key Responsibilities:
- Design, develop, and maintain scalable and reliable data pipelines
- Develop data processing solutions using PySpark
- Build and manage ETL/ELT pipelines using AWS Glue
- Orchestrate and schedule data workflows using Apache Airflow
- Work with Amazon S3 and Apache Iceberg for data storage and management
- Develop and optimize data solutions using Amazon Redshift
- Work with PostgreSQL for data storage, querying, and data integration
- Monitor data pipelines and proactively identify and resolve issues
- Troubleshoot pipeline failures and data-related issues
- Perform performance optimization of data pipelines and queries
- Ensure data quality, reliability, scalability, and security
- Contribute to data architecture and technical design decisions
- Collaborate with cross-functional teams to understand business and data requirements
Required Skills & Experience:
- Strong experience in AWS Data Engineering
- Hands-on experience with PySpark
- Strong experience with AWS Glue
- Experience in Apache Airflow for workflow orchestration
- Good experience with Amazon S3 and Apache Iceberg
- Experience with Amazon Redshift
- Strong knowledge of PostgreSQL
- Strong experience in data pipeline design and development
- Experience in pipeline monitoring, troubleshooting, and performance optimization
- Good understanding of data engineering concepts and data architecture
- Solid analytical and problem-solving skills
If interested, please share your updated CV to [HIDDEN TEXT]