Responsibilities-

1. Data Engineering & Pipeline Development

Monitor and optimize compute usage across workloads. Evaluate storage patterns, caching techniques, and data partitioning. Design, build, and maintain scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).

Architect data ingestion frameworks to process structured, semi-structured, and unstructured data.

Develop re-usable frameworks for data quality, data validation, and exception handling.

1. Data bricks & Cloud Platform Expertise

• Optimize Data bricks clusters, jobs, and queries for performance and cost efficiency.
- Manage cluster configurations, auto scaling, job scheduling, and deployment workflows.
- Implement best practices for notebook modularization, CI/CD, and version control.

1. Data Architecture & Modelling

• Design end-to-end data models including lakehouse architecture, medallion layers (bronzesilvergold).
- Collaborate with Data Science team to prepare training datasets and feature engineering pipelines.
- Implement governance, cataloguing, and lineage using cloud-native tools.

1. Performance & Cost Optimization

• Recommend architecture and platform-level improvements to reduce cost.




1. Collaboration & Documentation

• Work closely with cross-functional teams including Data Science, Analytics, Business, and IT Infra.
- Maintain detailed documentation for pipelines, data flows, and technical designs.
- Support production deployments, monitoring, and troubleshooting.

Educational Qualifications Graduation/Post Graduation

Competencies Required:

- Strong hands-on knowledge of Databricks, Spark, Delta Lake, and PySpark.
- Experience in architecting and scaling cloud-based pipelines in Azure • Strong SQL expertise and understanding of distributed data processing.
- Experience with source versioning (Git), CI/CD, and DevOps workflows. • Proficiency with REST APIs, Databricks SQL Warehouses, and Unity Catalog.
- Positive understanding of data warehousing concepts, lakehouse patterns, and metadata management.
- Exposure to ML model deployment workflows
- Intermediate proficiency in Power BI (dataset preparation, modeling, dashboards).

Experience - 5+ years of experience as a Data Engineer or in a similar role.

📌 Senior/Lead Data Engineer (Mumbai)
🏢 ICICI Lombard
📍 Mumbai