4+ years of handson experience in development, troubleshooting in AWS Glue
Know about using AWS glue in streaming solution.Integration of Glue with AWS other services.Data Quality implementation using Glue.Using Spark on Hadoop setting
SN
Responsibility of / Expectations from the Role
1
Able to implement efficient, scalable and reliable data integration solution using AWS Glue Service. Design and develop ETL workflows using AWS Glue components like Crawlers, Jobs and triggers. He will be able to write and optimize ETL Scripts in PySpark, configure data mapping, transformation using Glue visual interface or code editor.
Working experience in processing semi-structured data like json, XML using Glue.
2
Know about metadata management using Glue Data Catalog & Crawler.
3
Know to develop custom transformation using AWS Glue development libraries.
4
Managing AWS Glue Costs and DPU Capacity planning.
5
Able to develop Data Security and compliance pattern by applying AWS Glues security best practices like encrypting data in transit and at rest. Managing Access control and auditing data access and usage.