18 Nov
|
Nous Info Solution
|
Bengaluru
18 Nov
Nous Info Solution
Bengaluru
- Design, develop, and maintain scalable data pipelines using Azure Databricks and Apache Spark.
- Build and optimize ETL/ELT workflows using PySpark, Spark SQL, and Python.
- Implement Lakehouse architecture using Delta Lake and Medallion Architecture (Bronze, Silver, and Gold layers).
- Develop and manage Delta Live Tables (DLT) and structured streaming pipelines.
- Work with Unity Catalog for data governance, access control, and metadata management.
- Implement data transformation, cleansing, validation, and integration processes.
- Optimize Spark jobs, cluster configurations, partitioning, caching, and query performance.
- Work with Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, and other cloud services.
- Implement incremental data processing, CDC, and Slowly Changing Dimensions (SCD Type 1 and Type 2).
- Build and manage workflows using Databricks Workflows and Jobs.
- Implement CI/CD pipelines and deployment automation for Databricks solutions.
- Collaborate with data architects, analysts, and business teams to deliver reliable data solutions.
- Ensure data quality, security, monitoring, and performance optimization.
📌 Databricks Lead (Bengaluru)
🏢 Nous Info Solution
📍 Bengaluru