- Design and architect scalable data processing solutions using Databricks and PySpark (Spark Streaming not DLT).
- Develop and maintain data pipelines, ensuring data quality and reliability.
- Implement data orchestration workflows using Azure Event Hubs and Azure Event Grid.
- Collaborate with data engineers, data scientists, and other stakeholders to understand data requirements and deliver solutions.
- Optimize and tune Spark queries for performance and efficiency.
- Ensure data security and compliance with industry standards and best practices.
- Provide technical guidance and mentorship to junior team members.
- Stay updated with the latest trends and advancements in Databricks, PySpark,and Azure services.