Skill Set - Azure Databricks, Pyspark, ADF, SQL, DevOps
Experience - 6-12 yrs
Location - Bangalore/Chennai
Notice Period - Immediate Joiner
Job Descriptions
- Design, develop, and optimize scalable batch and streaming data pipelines using Azure Databricks.
- Develop data processing frameworks using PySpark and Spark SQL.
- Implement Delta Lake-based storage solutions to ensure high reliability and data consistency.
- Build and maintain Bronze, Silver, and Gold data layers within a Lakehouse architecture.
- Ingest data from APIs, databases, Azure storage, and streaming platforms.
- Develop scalable data models supporting analytics and reporting requirements.
- Implement data quality, validation, and reconciliation processes.
- Monitor and optimize Spark workloads for performance and cost efficiency.
- Establish and support data governance practices through Unity Catalog and related solutions.
- Orchestrate and integrate data workflows using Azure Data Factory, Airflow, and Databricks Workflows.
- Collaborate with business and technical stakeholders to gather requirements and deliver enterprise-grade data solutions.