Job Summary

We are seeking a skilled Databricks Developer to design, develop, and optimize scalable data pipelines and analytics solutions using Apache Spark, Python (PySpark), and SQL within up-to-date cloud environments. The ideal candidate will have hands-on experience working with Databricks, Delta Lake, and cloud data platforms, and will be responsible for building high-performance data processing workflows that support data-driven decision-making.

Key Responsibilities

Pipeline Development
- Design, develop, and maintain ETL/ELT data pipelines using PySpark and SQL in Databricks notebooks.
- Process large-scale datasets and ensure reliable and efficient data transformations.

Architecture Design
- Implement data lake and data warehouse architectures using Databricks Delta Lake and Delta Live Tables.
- Build and manage Medallion Architecture (Bronze, Silver, Gold layers) for structured data processing.

Performance Optimization
- Optimize Spark jobs and queries for performance, scalability, and cost-efficiency.
- Manage cluster configurations, partitioning strategies,



and caching mechanisms.

Cloud Integration
- Integrate Databricks solutions with cloud services such as:
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS) Gen2
- AWS S3 or other cloud storage platforms

Data Governance & Quality
- Implement data quality checks, validation frameworks, and monitoring processes.
- Ensure data security, encryption, masking, and lineage tracking.

Workflow Automation
- Build and manage automated workflows and scheduling using Databricks Jobs, Airflow, or CI/CD pipelines.
- Integrate with DevOps tools such as Azure DevOps or Jenkins for continuous integration and deployment.

Collaboration
- Work closely with data engineers, analysts, and business stakeholders to translate business requirements into scalable data solutions.
- Participate in Agile development processes including sprint planning and technical discussions.

Required Skills & Qualifications

Core Te

📌 Databricks Developer (Chennai)
🏢 Kumaran Systems
📍 Chennai