Roles and Responsibilities :

- Design, develop, test, deploy and maintain large-scale data pipelines using Azure Data Factory (ADF) to integrate various data sources into a centralized platform.
- Collaborate with cross-functional teams to gather requirements for data processing needs and design solutions that meet business objectives.
- Develop complex SQL queries to extract insights from large datasets stored in Azure Databricks.
- Troubleshoot issues related to ADF pipeline failures and optimize performance for improved efficiency.

Job Requirements :

- 6-15 Years of in Big Data & Data related technology experience
- Expert level understanding of distributed computing principles
- Expert level knowledge and experience in Apache Spark Hands on programming with Python Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop
- Experience with building stream-processing systems,



using technologies such as Apache Storm or Spark-Streaming
- Experience with messaging systems, such as Kafka or RabbitMQ Good understanding of Big Data querying tools, such as Hive, and Impala
- Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files
- Positive understanding of SQL queries, joins, stored procedures, relational schemas
- Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
- Knowledge of ETL techniques and frameworks Performance tuning of Spark Jobs
- Experience with native Cloud data services AWS Databricks, Azure databricks
- Ability to lead a team efficiently
- Experience with designing and implementing Big data solutions
- Practitioner of AGILE methodology

📌 6-15 Years of in Big Data & Data related technology experience (Pune)
🏢 EPAM Systems
📍 Pune