Must-Have • Design, build, and maintain scalable and efficient data pipelines using Databricks. Integrate data from various sources (for eg. Azure storage account, SFTP, Fabric) using Apache Spark within Databricks.
- Implement processes and checks to ensure the accuracy and completeness of data.
- Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutions.
- Monitor data pipelines for issues and resolve them promptly to maintain data flow integrity.
- Experience with Databricks-specific tools and features.
- Proficiency in programming languages such as Python or Scala for data processing on Databricks • Experience with Azure cloud platform.
- Strong understanding of SQL.
- Experience with ETL and processes, Unity Catalog.
Valuable-to-Have Certifications in Databricks or related cloud technologies
SN Responsibility of / Expectations from the Role
1 Design, build, and maintain scalable and efficient data pipelines using Databricks. Integrate data from various sources (for eg. Azure storage account, SFTP, Fabric) using Apache Spark within Databricks.
2 Implement processes and checks to ensure the accuracy and completeness of data.
3 Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutions
4 Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutions.
5 Monitor data pipelines for issues and resolve them promptly to maintain data flow integrity.