Build and operationalize robust data engineering pipelines using AWS Glue PySpark and Delta Lake to create governed high-quality enterprise datasets.
KEY RESPONSIBILITIES
- Develop AWS Glue / PySpark transformations from Raw to Refined datasets in Amazon S3.
- Implement schema enforcement type conversion PII tokenization deduplication and merges.
- Develop Delta Lake pipelines and when required Gold datasets and Redshift loads.
- Orchestrate jobs using Airflow or AWS Step Functions with completeness and failure checks.
- Register datasets in Unity Catalog and support governance metadata and consumer access.
- Perform reconciliation performance tuning unit testing and production troubleshooting.
- Prepare CI/CD artifacts deployment configuration runbooks and technical documentation.
Requirements
REQUIRED SKILLS
- Python PySpark Spark SQL and strong SQL
- AWS Glue Amazon S3 Delta Lake and Databricks
- ETL/ELT data modelling and data-quality validation
- Airflow or AWS Step Functions and failure recovery
- Git CI/CD data reconciliation and production support
PREFERRED EXPERIENCE
Unity Catalog Amazon Redshift Immuta ServiceNow integration and regulated-data experience.
Required Skills:
Python PySpark Spark SQL and robust SQL AWS Glue Amazon S3 Delta Lake and Databricks ETL/ELT data modelling and data-quality validation Airflow or AWS Step Functions and failure recovery Git CI/CD data reconciliation and production support
📌 Senior Data Engineer (Hyderabad)
🏢 RADcube - A NLogix
📍 Hyderabad
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.