- Develop and optimize data solutions using Delta Sharing and Databricks Unity Catalog Administration.
- Implement and manage Databricks CLI and Delta Live Pipelines for efficient data processing.
- Utilize Structured Streaming for real-time data processing and analysis.
- Apply risk management strategies to ensure data integrity, security, and compliance.
- Integrate Apache Airflow for workflow orchestration and automation.
- Leverage Amazon S3 and Amazon Redshift for scalable data storage and retrieval.
- Develop and maintain Python scripts to automate data processing tasks.
- Utilize Databricks SQL and Delta Lake for data querying, transformation, and storage optimization.
- Design and implement Databricks Workflows to automate data pipelines.
- Develop PySpark applications to process and transform large datasets.
- Collaborate with cross-functional teams to understand requirements and deliver effective data solutions.
- Monitor and troubleshoot data workflows to ensure smooth operations.
- Stay updated with the latest industry trends and technologies to improve data solutions.
Qualifications:
- Solid experience in Delta Sharing and Databricks Unity Catalog Administration.
- Proficiency in Databricks CLI and Delta Live Pipelines.
- Good understanding of Structured Streaming and Risk Management.
- Experience with Apache Airflow, Amazon S3, and Amazon Redshift.
- Proficiency in Python and PySpark.
- Strong problem-solving and analytical skills.