Key Responsibilities

· Data Pipeline Development:
Design, build, and maintain scalable data pipelines and architectures using Databricks Lakehouse Platform (Delta Lake, Spark, Spark SQL, Auto Loader, and MLflow)
Implement complex ETL/ELT solutions across bronze, silver, and gold layers.
Build reusable frameworks for ingestion, transformation, and monitoring.
Databricks Platform Engineering:
Develop and optimize Databricks notebooks, workflows, and job clusters.
Implement best practices for Delta Lake: optimization, compaction, schema enforcement, time travel.
Troubleshoot performance issues and fine-tune Spark jobs for efficiency.
Cloud Integration (Azure / AWS):
Integrate Databricks with cloud storage (ADLS/S3).
Implement security, secrets, and networking configurations.
Data Quality, Observability & Governance:
Build data validation frameworks and error-handling mechanisms.
Work with Unity Catalog for cataloging, governance, and access control.
Ensure data compliance, security standards, and audit processes.
DevOps & CI/CD:
Maintain Git-based development workflows.
Develop CI/CD pipelines for Databricks workflows using Git, and orchestration tools like Apache Airflow or Databricks Workflows.
Collaboration, Leadership & Guidance:




Collaborate with architects, analysts, and business stakeholders to refine requirements.
Mentor junior engineers and conduct code reviews.
Participate in solution design and architecture discussions.

Skills and Qualifications

· Bachelor’s / Master’s degree in computer science, computer engineering, or relevant field.

· Overall 3+ years of experience, at least 2 years of experience in Databricks Data Engineering.

· Robust, hands-on experience with the Databricks workspace, Delta Lake, Delta Live Tables (DLT), and Unity Catalog.

· Proficiency in Apache Spark, PySpark, and distributed computing principles.

· Advanced proficiency in Python and SQL.

· Hands-on experience with cloud settings such as AWS, Azure.

· Experience with Git, CI/CD pipelines, and infrastructure-as-code tools (Terraform, CloudFormation, or Databricks Asset Bundles).

· Experience with ETL processes: Extracting, transforming, and loading data.

· Strong analytical and problem-solving skills.

· Positive communication and collaboration skills.

· Experience with data security and governance principles.

· Understanding of data visualization techniques.

📌 Databricks Data Engineer Pune (India)
🏢 Emergys
📍 India