Key Responsibilities:
- Design, develop, and optimize ETL/ELT pipelines using Databricks
- Develop data solutions using PySpark, Spark SQL, and Databricks notebooks
- Work with large-scale structured and unstructured datasets
- Integrate data from multiple sources into data lakes and data warehouses
- Monitor, troubleshoot, and improve data pipeline performance
- Collaborate with business stakeholders, architects, and data analysts
- Implement CI/CD and data governance best practices
- Databricks with any cloud.