Job Summary:

We are looking for a Senior Data Engineer with 6+ years of experience in designing, developing, and supporting scalable data pipelines. The candidate should have strong hands-on experience in Python, PySpark, AWS data services, Spark, EMR, Delta Lake, Kinesis, and workflow orchestration.

The role involves building and maintaining ETL/ELT pipelines, working with large datasets, optimizing data processing, and quickly resolving data-related issues.

Key Responsibilities:

- Design, develop, and maintain scalable and reliable data pipelines using Python and PySpark.
- Build and manage ETL/ELT processes for large and complex datasets.
- Develop data processing solutions using Apache Spark and AWS EMR.
- Implement serverless data workflows using AWS Lambda and Step Functions.
- Work with AWS S3, Glue, Athena, Redshift, and other AWS data services.
- Build and maintain real-time and streaming data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
- Work with Delta Lake for reliable data storage and processing.
- Perform data modelling, data transformation, and performance optimization.
- Monitor data pipelines and troubleshoot failures and data quality issues.
- Handle and resolve data hydration-related issues quickly and effectively.
- Implement best practices for code quality, version control, and deployment.
- Work with Git and Bitbucket for source code management and CI/CD processes.




- Collaborate with cross-functional teams to understand requirements and deliver data solutions on time.
- Support production data pipelines and resolve issues within agreed timelines.

Mandatory Skills:

- 6+ years of experience in Data Engineering.
- Strong hands-on experience with Python.
- Robust experience with PySpark and Apache Spark.
- Hands-on experience with AWS EMR.
- Hands-on experience with AWS Lambda.
- Hands-on experience with AWS Step Functions.
- Experience with Delta Lake.
- Experience with Amazon Kinesis Data Streams and Kinesis Data Firehose.
- Strong understanding of ETL/ELT processes and data pipelines.
- Experience working with large datasets and distributed data processing.
- Strong troubleshooting and production support skills.
- Experience with AWS S3 and other AWS data services.

Preferred Skills:

- Experience with AWS Glue, Athena, and Redshift.
- Experience with Snowflake and cloud-based data warehousing.
- Knowledge of data modelling and performance optimization.
- Experience with CI/CD processes.
- Hands-on experience with Git and Bitbucket.
- Experience in monitoring and troubleshooting production data pipelines.
- Good understanding of data quality and data validation processes.
- Strong communication and collaboration skills.
- Ability to work independently and deliver solutions within tight timelines.

📌 Aws Data Engineer (Bengaluru)
🏢 Qentelli
📍 Bengaluru