01 Aug
|
Important Company of the Sector
|
Hyderabad
01 Aug
Important Company of the Sector
Hyderabad
Job Title: Lead AWS PySpark Engineer
Location: Hyderabad
Experience: 9+ Years
Job Overview
We are seeking a highly skilled Lead AWS PySpark Engineer to design, develop, and optimize large -scale data processing pipelines on AWS. The ideal candidate will have strong experience in PySpark, distributed data processing, and AWS data services, along with the ability to lead technical initiatives and mentor data engineers.
Requirements
Key Responsibilities
- Design, build, and maintain scalable data pipelines using PySpark on AWS.
- Lead the development of ETL/ELT workflows for processing large volumes of structured and unstructured data.
- Architect and optimize data solutions using AWS services such as S3, Glue, Athena.
- Collaborate with data scientists, analysts, and product teams to deliver high -quality data solutions.
- Implement data quality checks, monitoring, and performance optimization for big data pipelines.
- Lead code reviews, enforce best practices, and mentor junior engineers.
- Work closely with DevOps teams to implement CI/CD pipelines and automated deployments.
- Ensure compliance with data governance, security, and best practices.
Required Skills
- 9+ years of experience in Data Engineering / Big Data Development.
- Strong expertise in PySpark and Apache Spark.
- Hands -on experience with AWS ecosystem (S3, Glue, EMR, Lambda, Redshift,
Athena).
- Proficiency in Python and SQL.
- Experience with data pipeline orchestration tools (Airflow, Step Functions, etc.).
- Solid knowledge of distributed computing and big data processing.
- Experience with data modeling, performance tuning, and query optimization.
- Familiarity with CI/CD tools, Git, and Agile development practices.