Key Responsibilities

· Data Pipeline Development:

- Design, build, and maintain scalable data pipelines and architectures using Databricks Lakehouse Platform (Delta Lake, Spark, Spark SQL, Auto Loader, and MLflow)
- Implement complex ETL/ELT solutions across bronze, silver, and gold layers.
- Build reusable frameworks for ingestion, transformation, and monitoring.
- Databricks Platform Engineering:
- Develop and optimize Databricks notebooks, workflows, and job clusters.
- Implement best practices for Delta Lake: optimization, compaction, schema enforcement, time travel.
- Troubleshoot performance issues and fine-tune Spark jobs for efficiency.
- Cloud Integration (Azure / AWS):
- Integrate Databricks with cloud storage (ADLS/S3).
- Implement security, secrets, and networking configurations.
- Data Quality, Observability & Governance:
- Build data validation frameworks and error-handling mechanisms.
- Work with Unity Catalog for cataloging, governance, and access control.
- Ensure data compliance, security standards, and audit processes.
- DevOps & CI/CD:
- Maintain Git-based development workflows.
- Develop CI/CD pipelines for Databricks workflows using Git, and orchestration tools like Apache Airflow or Databricks Workflows.
- Collaboration, Leadership & Guidance:




- Collaborate with architects, analysts, and business stakeholders to refine requirements.
- Mentor junior engineers and conduct code reviews.
- Participate in solution design and architecture discussions.

Skills and Qualifications

· Bachelor’s / Master’s degree in computer science, computer engineering, or relevant field.

· Overall 3+ years of experience, at least 2 years of experience in Databricks Data Engineering.

· Strong, hands-on experience with the Databricks workspace, Delta Lake, Delta Live Tables (DLT), and Unity Catalog.

· Proficiency in Apache Spark, PySpark, and distributed computing principles.

· Advanced proficiency in Python and SQL.

· Hands-on experience with cloud environments such as AWS, Azure.

· Experience with Git, CI/CD pipelines, and infrastructure-as-code tools (Terraform, CloudFormation, or Databricks Asset Bundles).

· Experience with ETL processes: Extracting, transforming, and loading data.

· Strong analytical and problem-solving skills.

· Positive communication and collaboration skills.

· Experience with data security and governance principles.

· Understanding of data visualization techniques.

📌 Databricks Data Engineer (Pune)
🏢 Emergys
📍 Pune