- Design, develop, and optimize data pipelines using Azure Databricks.
- Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions.
- Implement data transformation and processing tasks using PySpark.
- Ensure data quality and integrity across various data sources.
- Provide technical guidance and mentorship to junior team members.
Additional Responsibilities
- Minimum Qualifications Bachelor's degree in Computer Science, Information Technology, or a related field.
- Proven experience with Azure Databricks, with a minimum of 2 years in data engineering.
- Strong understanding of data processing frameworks and cloud technologies.
- Preferred Qualifications Experience with PySpark and its application in big data environments.
- Azure Databricks certification.
- Track record of successfully delivering data solutions in a consulting workplace.
- Familiarity with data governance and security best practices.
- Excellent problem-solving skills and ability to work independently.