Responsibilities:
• Design, develop, and maintain scalable data pipelines using Databricks and Azure Data Factory.
• Implement data processing solutions using Spark and Delta Tables to optimize performance and reliability.
• Collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets.
• Manage and optimize data storage solutions in Azure Data Lake and ensure data accessibility through Unity Catalog.
• Monitor and troubleshoot data pipeline performance and implement improvements as necessary.
• Document data processes and maintain clear communication with stakeholders regarding project status and challenges.
• Stay updated with the latest industry trends and technologies to continuously improve data engineering practices. Mandatory
Skills:
• Strong proficiency in SQL for data manipulation and querying.
• Expertise in Python for data processing and automation.
• Hands-on experience with Databricks is a deal breaker.
• Familiarity with Azure Data Factory for orchestrating data workflows.
• Experience with Spark for big data processing.
• Knowledge of Delta Tables for productive data storage and management.
• Understanding of Unity Catalog for data governance and management.
• Experience with Azure Data Lake for data storage solutions.
• Familiarity with Azure Synapse for data integration and analytics. Preferred
Skills:
• Experience with data modeling and ETL processes.
• Knowledge of data warehousing concepts and best practices.
• Familiarity with machine learning concepts and tools.
• Experience with version control systems such as Git.
• Strong problem-solving skills and attention to detail.