As a Lead Data Engineer specializing in Databricks, you will be a key player in designing, developing, and optimizing our enterprise transportation client's next-generation data platform. You will lead a team of data engineers, providing technical guidance, mentorship, and ensuring the scalable, and high-performance data solutions.

Key Responsibilities:

- Technical Leadership:
- Lead the design, development, and implementation of scalable and reliable data pipelines using Databricks, Spark, and other relevant technologies
- Define and enforce data engineering best practices, coding standards, and architectural patterns
- Provide technical guidance and mentorship to junior and mid-level data engineers
- Conduct code reviews and ensure the quality, performance, and maintainability of data solutions

- Databricks Expertise:

- Architect and implement data solutions on the Databricks platform, including Databricks Lakehouse, Delta Lake, and Unity Catalog
- Optimize Spark workloads for performance and cost efficiency on Databricks
- Develop and manage Databricks notebooks, jobs, and workflows
- Proficiently use Databricks features such as Delta Live Tables (DLT), Photon, and SQL Analytics

- Pipeline Development & Operations:

- Develop, test, and deploy robust ETL/ELT pipelines for data ingestion, transformation, and loading from various sources (e.g., relational databases, APIs, streaming data)
- Implement monitoring, alerting, and logging for data pipelines to ensure operational excellence
- Troubleshoot and resolve complex data-related issues

- Collaboration & Communication:

- Work closely with cross-functional teams including product managers, data scientists, and software engineers
- Communicate complex technical concepts clearly to both technical and non-technical stakeholders




- Stay updated with industry trends and emerging technologies in data engineering and Databricks

Must Have Skills:

- Experience:
- 8+ years of experience in data engineering, with at least 2-3 years in a lead or senior capacity
- Proven experience designing and building large-scale data platforms

- Primary Skills - Databricks:

- Extensive hands-on experience with Databricks platform, including Databricks Workspace, Spark on Databricks, Delta Lake, and Unity Catalog
- Robust proficiency in optimizing Spark jobs and understanding Spark architecture
- Experience with Databricks features like Delta Live Tables (DLT), Photon, and Databricks SQL Analytics

- Programming:

- Expertise in Python (PySpark) is essential
- Strong proficiency in SQL

- Data Warehousing/Lakes:

- Deep understanding of data warehousing concepts, dimensional modeling, and data lake architectures
- Experience with various data storage formats (Parquet, ORC, JSON, CSV)

- Cloud Platforms:

- Experience with at least one major cloud platform (AWS, Azure, or GCP) and their data-related services (e.g., S3, ADLS, GCS, EC2, Azure VMs, Google Compute Engine)

- Tools & Technologies:

- Experience with version control systems (Git)
- Familiarity with CI/CD pipelines for data solutions
- Knowledge of workflow orchestration tools (e.g., Apache Airflow, Databricks Workflows)

- Education:

- Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field

Preferred Skills:

- Experience with stream processing technologies (e.g., Kafka, Kinesis) is a plus
- Contributions to open-source data projectsExperience with Scala or Java is a plus
- Databricks certifications (e.g., Databricks Certified Data Engineer Associate/Professional)
- Experience with MLOps and integrating data pipelines with machine learning workflows

📌 Data Lead (Maharashtra)
🏢 BridgeNext
📍 Maharashtra