20 Nov
|
Durapid Technologies
|
Bengaluru
20 Nov
Durapid Technologies
Bengaluru
We are looking for a Senior AWS Data Engineer with solid expertise in building, optimising and maintaining scalable data pipelines across AWS environments. The ideal candidate should have deep hands-on experience with Python, SQL, PySpark, Apache Spark and AWS data services, along with the ability to troubleshoot complex workloads and improve pipeline performance, reliability and cost efficiency.
The role requires a technically strong individual contributor who can work independently with clients and engineering teams, provide technical guidance, and deliver reliable data foundations for analytics and AI-driven solutions.
Key Responsibilities
- Design, develop and maintain scalable ETL/ELT pipelines using AWS services.
- Build reusable data engineering components using Python, SQL and PySpark.
- Optimise complex SQL queries, Spark jobs and distributed data processing workloads.
- Analyse Spark execution plans, data skew, shuffle operations, memory utilisation and resource consumption.
- Improve partitioning, file sizing, compression, caching and join strategies.
- Tune AWS Glue, Amazon Redshift, Amazon Athena and other AWS processing services.
- Implement automated testing, data validation, monitoring, exception handling and recovery mechanisms.
- Troubleshoot production issues, perform root-cause analysis and implement preventive solutions.
- Establish performance benchmarks and measure improvements in runtime, throughput and cloud costs.
- Apply data modelling, metadata management,
security and governance best practices.
- Maintain technical documentation, operational runbooks and engineering standards.
- Mentor engineers and collaborate with clients, platform teams and business stakeholders.
Mandatory Skills
- Strong hands-on experience in AWS data engineering: S3, Glue, Redshift, Athena and Lambda.
- Advanced SQL with strong query optimisation and troubleshooting skills.
- Proficiency in Python and PySpark.
- Strong knowledge of Apache Spark and distributed data processing.
- Experience with Apache Airflow or equivalent orchestration tools.
- Understanding of data lakes, lakehouse architectures, data warehouses and dimensional modelling.
- Experience with Git, CI/CD, automated testing and production support.
- Practical experience in performance optimisation of large-scale data pipelines.
Good to Have
- Apache Kafka and/or Amazon Kinesis.
- Amazon Bedrock, LLMs, embeddings and vector databases.
- Excel/CSV ingestion, template-driven processing and schema evolution.
- Experience with columnar formats such as Parquet and Iceberg.
Soft Skills
- Strong communication, comprehension and articulation skills.
- Ability to work independently as a technical consultant and individual contributor.
- Strong stakeholder management, persuasion and problem-solving skills.
- Ability to guide, mentor and inspire engineering teams.
- Strong ownership, analytical thinking and client-facing capabilities.
📌 Senior AWS Data Engineer (Bengaluru)
🏢 Durapid Technologies
📍 Bengaluru