Job Summary

We are seeking a skilled Databricks Developer to design, develop, and optimize scalable data pipelines and analytics solutions using Apache Spark, Python (PySpark), and SQL within contemporary cloud environments. The ideal candidate will have hands-on experience working with Databricks, Delta Lake, and cloud data platforms, and will be responsible for building high-performance data processing workflows that support data-driven decision-making.

Key Responsibilities

Pipeline Development
- Design, develop, and maintain ETL/ELT data pipelines using PySpark and SQL in Databricks notebooks.
- Process large-scale datasets and ensure reliable and efficient data transformations.

Architecture Design
- Implement data lake and data warehouse architectures using Databricks Delta Lake and Delta Live Tables.
- Build and manage Medallion Architecture (Bronze, Silver, Gold layers) for structured data processing.

Performance Optimization
- Optimize Spark jobs and queries for performance, scalability, and cost-efficiency.
- Manage cluster configurations, partitioning strategies,



and caching mechanisms.

Cloud Integration
- Integrate Databricks solutions with cloud services such as:
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS) Gen2
- AWS S3 or other cloud storage platforms

Data Governance & Quality
- Implement data quality checks, validation frameworks, and monitoring processes.
- Ensure data security, encryption, masking, and lineage tracking.

Workflow Automation
- Build and manage automated workflows and scheduling using Databricks Jobs, Airflow, or CI/CD pipelines.
- Integrate with DevOps tools such as Azure DevOps or Jenkins for continuous integration and deployment.

Collaboration
- Work closely with data engineers, analysts, and business stakeholders to translate business requirements into scalable data solutions.
- Participate in Agile development processes including sprint planning and technical discussions.

Required Skills & Qualifications

Core Te

📌 Databricks Developer (Hyderabad)
🏢 Kumaran Systems
📍 Hyderabad