Role Overview: As a Databricks Consultant, you will be responsible for developing and implementing data engineering solutions on the Azure Databricks platform.
Your role involves designing and maintaining scalable data pipelines, performing data transformations using Spark, and ensuring data is properly structured within a Delta Lake workplace for downstream analytics.
Experience:
- 3–5 years of experience in data engineering, with a strong emphasis on AzureDatabricks.
- Proficiency in PySpark or SparkSQL for data transformation.
- Experience with Azure Data Lake Storage (ADLS Gen2) and Azure Data Factory (ADF).
- Strong foundation in SQL and relational databases (SQL Server, Oracle, or PostgreSQL).
Key Responsibilities: Technical Skills:
- Develop and maintain ETL/ELT pipelines using AzureDatabricks to move and transform data across diverse sources.
- Implement data processing workflows using PySpark and SparkSQL within Databricks notebooks.
- Manage and organize data within Delta Lake,
utilizing features like Delta tables and schema enforcement.
- Use Azure Data Factory (ADF) to orchestrate Databricks notebooks and workflows.
- Integrate various data sources (SQL Server, Oracle, etc.) into the Azure Databricks ecosystem.
- Support data cataloging and documentation efforts to ensure data discoverability.
- Troubleshoot and resolve issues related to Spark jobs, data pipelines, and data quality.
Soft Skills:
- Strong problem-solving and analytical abilities.
- Ability to work effectively within agile, cross-functional teams.
- Strong communication skills for technical documentation and team collaboration.
Preferred Qualifications:
- Bachelor’s degree in Computer Science or a related field.
- Databricks Certified Data Engineer Associate is highly desirable.