Job Description
• Should have good understanding and real time experience on Hadoop ecosystem (HDFS, Map Reduce, YARN, Hive, Sqoop, Spark) and Azure cloud (ADLS, ADFS, ADF)
• Should have real time experience by using various file formats (csv, parquet, text file, avro, sequential file)
• Should have Hands-on experience with hive internal, external tables, hive join techniques, partitions, bucketing and hive optimization techniques
• Should have hands-on experience in developing Spark applications using Data frame API, Spark-SQL in Data bricks for data extraction, transformation, and aggregation from multiple file formats for Analyzing & transforming the data using Scala and Python to uncover insights into the customer usage patterns.
• Should have hands-on experience Extract Transform and Load data from sources Systems (On-prem) to Azure Data Storage services using a combination of Azure Data factory, T-SQL, Spark SQL.
• Should have hands-on Experience metadata driven ingestion pipeline framework by using Azure Data Factory Should have valuable analytics and writing SQL queries to consume data and generate reports using Power BI Qualifications BTech/MTech Range Of Year Experience-Min Year 5 Range Of Year Experience-Max Year 7