Roles and Responsibilities
- Design, develop, and maintain scalable data pipelines using AWS Glue (Spark, Python) and Delta Lake for efficient ETL processing.
- Model and transform data across different layers (raw, staging, processed) to support analytics and reporting requirements.
- Orchestrate complex data workflows using AWS Step Functions, coordinating Glue jobs and Lambda functions.
- Manage data storage solutions, primarily using Amazon S3 and relational databases such as Oracle.
- Expose structured and curated data for analytics and reporting via AWS Athena.
- Implement and maintain CI/CD processes for ETL jobs and data pipeline deployments using GitHub.
- Enable and support data access for business intelligence tools (e.g., Power BI).
- Collaborate with cross-functional teams to understand business data needs and deliver effective solutions.
- Operate within secure development environments (e.g., Laptop/VDI) to access required data and toolsets.
Skills & Experience
- Demonstrated experience building data pipelines on AWS (Glue, S3, Athena, Step Functions).
- Solid background in ETL development with PySpark and Python.
- Experience with Delta Lake or similar data lake architectures.
- Solid understanding of data modelling, data transformation, and performance optimization.
- Hands-on experience with version control (GitHub) and CI/CD for data engineering workflows.
- Familiarity with business intelligence/reporting tools integration (e.g., Power BI).
- Experience working in secure or remote development environments.
Skills
Must Have
- AWS Glue (Spark, Python) for ETL development
- Amazon S3 for data storage and management
- AWS Step Functions for workflow orchestration
- AWS Athena for data querying and analytics
- Delta Lake or similar table format technologies
- GitHub for CI/CD and script versioning
- Data modelling and transformation expertise
- SQL proficiency
Nice to Have
- Experience wi
📌 AWS Data Engineer (Noida)
🏢 Topcoder
📍 Noida
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.