- Design, develop, and maintain scalable data pipelines using Databricks and Azure Data Factory.
- Implement data processing solutions using Spark and Delta Tables to optimize performance and reliability.
- Collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets.
- Manage and optimize data storage solutions in Azure Data Lake and ensure data accessibility through Unity Catalog.
- Monitor and troubleshoot data pipeline performance and implement improvements as necessary.
- Document data processes and maintain clear communication with stakeholders regarding project status and challenges.
- Stay updated with the latest industry trends and technologies to continuously improve data engineering practices.
Mandatory Skills:
- Strong proficiency in SQL for data manipulation and querying.
- Expertise in Python for data processing and automation.
- Hands-on experience with Databricks is a deal breaker.
- Familiarity with Azure Data Factory for orchestrating data workflows.
- Experience with Spark for big data processing.
- Knowledge of Delta Tables for productive data storage and management.
- Understanding of Unity Catalog for data governance and management.
- Experience with Azure Data Lake for data storage solutions.
- Familiarity with Azure Synapse for data integration and analytics.
Preferred Skills:
- Experience with data modeling and ETL processes.
- Knowledge of data warehousing concepts and best practices.
- Familiarity with machine learning concepts and tools.
- Experience with version control systems such as Git.
- Strong problem-solving skills and attention to detail.