Must Have Skills: SQL, PySpark, Databricks, AWS, Python
Positive To Have Skills: PySpark, AWS CloudWatch, AWS S3, Streaming
External Description
This is a technical role.
No shift timings :
- Experience in AWS Databricks Data pipelines Development - DLT, Streaming & Data ingestion patterns, Unity Catalogue migration
- Independently develop AWS data extraction and integration code modules for one-time historical, incremental data flows from Databricks Delta Lake using PySpark, Python, SQL, AWS Databricks, and S3.
- Develop scalable data ingestion pipelines as needed
- Implement advanced transformation logics, schedule, orchestrate, and validate pipelines
- Perform data validation, data reconciliation
- Deployment of Components
- Perform unit testing and document test results.