Detailed JD

AWS Data Engineer – Data Pipeline & Modelling

Roles and Responsibilities

- Design, develop, and maintain scalable data pipelines using AWS Glue (Spark, Python) and Delta Lake for effective ETL processing.

- Model and transform data across different layers (raw, staging, processed) to support analytics and reporting requirements.

- Orchestrate complex data workflows using AWS Step Functions, coordinating Glue jobs and Lambda functions.

- Manage data storage solutions, primarily using Amazon S3 and relational databases such as Oracle.

- Expose structured and curated data for analytics and reporting via AWS Athena.

- Implement and maintain CI/CD processes for ETL jobs and data pipeline deployments using GitHub.

- Enable and support data access for business intelligence tools (e.g., Power BI).

- Collaborate with cross-functional teams to understand business data needs and deliver effective solutions.

- Operate within secure development environments (e.g., Laptop/VDI) to access required data and toolsets.

Skills & Experience

- Demonstrated experience building data pipelines on AWS (Glue, S3, Athena, Step Functions).

- Strong background in ETL development with PySpark and Python.





- Experience with Delta Lake or similar data lake architectures.

- Solid understanding of data modelling, data transformation, and performance optimization.

- Hands-on experience with version control (GitHub) and CI/CD for data engineering workflows.

- Familiarity with business intelligence/reporting tools integration (e.g., Power BI).

- Experience working in secure or remote development environments.

Skills

Must Have

- AWS Glue (Spark, Python) for ETL development

- Amazon S3 for data storage and management

- AWS Step Functions for workflow orchestration

- AWS Athena for data querying and analytics

- Delta Lake or similar table format technologies

- GitHub for CI/CD and script versioning

- Data modelling and transformation expertise

- SQL proficiency

Nice to Have

- Experience with AWS Lambda for serverless transformation tasks

- Exposure to Oracle or other relational databases

- Experience with Power BI or similar reporting/visualization tools

- Familiarity with customer service or CRM data sources especially Salesforce & IVR

- Knowledge of secure workspace access (Laptop/VDI environments)

📌 AWS Data Engineer - Data Pipeline & Modelling (Noida)
🏢 Topcoder
📍 Noida