Job Description: DataBricks - Lead Programmer Analyst

We are looking for a highly self-motivated individual with DataBricks development as a Lead Programmer Analyst:

Required Skills & Qualifications:

- - Experience should have 5 to 7 Years of Data Engineering.
- Expert-level Apache Spark skills using PySpark (Scala a plus).
- Strong proficiency in SQL for data transformation and performance tuning.
- Solid experience with Delta Lake and the Lakehouse/medallion architecture.
- Hands-on experience with at least one major cloud platform (AWS, Azure, or GCP) and its data services.
- Experience with workflow orchestration (Databricks Workflows, Delta Live Tables, or Airflow).
- Strong understanding of data modeling, data warehousing, and ETL/ELT design patterns.
- Experience with Python for data engineering and automation.
- Familiarity with CI/CD, Git, and DevOps practices for data.
- Good understanding of data governance, security, and Unity Catalog.
- Robust problem-solving and communication skills.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Good To Have:

- - Databricks Certified Data Engineer Associate or Professional certification.
- Experience with streaming technologies (Structured Streaming, Kafka, Event Hubs, Pub/Sub).
- Exposure to machine learning workflows and MLflow.
- Experience with Infrastructure-as-Code (Terraform) and cloud cost optimization.




- Experience working in Agile delivery environments.
- Experience in data warehouse design and maintenance
- Experience in agile development processes using Jira and Confluence.
- Experience in cross-functional teams.

Key Responsibilities:

- - Design, develop, and maintain scalable ETL/ELT data pipelines on the Databricks Lakehouse Platform.
- Build and optimize Apache Spark (PySpark/Scala) jobs for batch and streaming data processing.
- Implement and manage Delta Lake tables, including partitioning, schema evolution, and performance tuning (Z-ordering, caching, file compaction).
- Develop and orchestrate workflows using Databricks Workflows, Delta Live Tables (DLT), and job scheduling tools such as Airflow.
- Implement the medallion architecture (bronze, silver, gold layers) for reliable, governed data.
- Integrate Databricks with cloud data services (AWS, Azure, or GCP) and source systems.
- Apply data governance and security using Unity Catalog, access controls, and lineage.
- Optimize cluster configuration, cost, and performance across workloads.
- Collaborate with data analysts, data scientists, and business stakeholders to deliver trusted datasets.
- Ensure data quality, testing, monitoring, and observability across pipelines.
- Contribute to CI/CD practices for data engineering (version control, automated deployment).

📌 DataBricks - Lead Programmer Analyst (India)
🏢 BILVANTIS TECHNOLOGIES
📍 India