Key Responsibilities

- Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.

- Build and optimize ETL/ELT processes for ingesting data from multiple source systems.

- Develop data transformation logic and implement business rules using Spark frameworks.

- Work closely with business stakeholders, data analysts, and architects to understand data requirements.

- Implement and manage Delta Lake architecture for reliable and high-performance data processing.

- Optimize Databricks jobs, notebooks, clusters, and Spark code for performance and cost efficiency.

- Develop and support data ingestion frameworks using Azure Data Factory (ADF) and other integration tools.

- Monitor production workloads, troubleshoot failures, and ensure data availability.

- Implement data quality checks, validations, and reconciliation processes.

- Ensure adherence to data governance, security, and compliance standards.

- Participate in code reviews, technical design discussions, and best practice implementation.

- Support production deployments and provide L2/L3 support for critical data pipelines.

📌 Databrick Engineer (Thane)
🏢 Workforce Solution
📍 Thane