Detailed JD

AWS Data Engineer – Data Pipeline & Modelling

Roles and Responsibilities
Design, develop, and maintain scalable data pipelines using AWS Glue (Spark, Python) and Delta Lake for effective ETL processing.
Model and transform data across different layers (raw, staging, processed) to support analytics and reporting requirements.
Orchestrate complex data workflows using AWS Step Functions, coordinating Glue jobs and Lambda functions.
Manage data storage solutions, primarily using Amazon S3 and relational databases such as Oracle.
Expose structured and curated data for analytics and reporting via AWS Athena.
Implement and maintain CI/CD processes for ETL jobs and data pipeline deployments using GitHub.
Enable and support data access for business intelligence tools (e.g., Power BI).
Collaborate with cross-functional teams to understand business data needs and deliver effective solutions.
Operate within secure development settings (e.g., Laptop/VDI) to access required data and toolsets.

Skills & Experience
Demonstrated experience building data pipelines on AWS (Glue, S3, Athena, Step Functions).
Robust background in ETL development with PySpark and Python.




Experience with Delta Lake or similar data lake architectures.
Solid understanding of data modelling, data transformation, and performance optimization.
Hands-on experience with version control (GitHub) and CI/CD for data engineering workflows.
Familiarity with business intelligence/reporting tools integration (e.g., Power BI).
Experience working in secure or remote development environments.

Skills

Must Have
AWS Glue (Spark, Python) for ETL development
Amazon S3 for data storage and management
AWS Step Functions for workflow orchestration
AWS Athena for data querying and analytics
Delta Lake or similar table format technologies
GitHub for CI/CD and script versioning
Data modelling and transformation expertise
SQL proficiency

Nice to Have
Experience with AWS Lambda for serverless transformation tasks
Exposure to Oracle or other relational databases
Experience with Power BI or similar reporting/visualization tools
Familiarity with customer service or CRM data sources especially Salesforce & IVR
Knowledge of secure workspace access (Laptop/VDI environments)

📌 Aws Data Engineer Data Pipeline & Modelling Noida (India)
🏢 Topcoder
📍 India