The Senior Tech Lead - Databricks leads the design, development, and implementation of advanced data solutions. Has To have extensive experience in Databricks, cloud platforms, and data engineering, with a proven ability to lead teams and deliver complex projects.
Responsibilities:
Lead the design and implementation of Databricks-based data solutions.
Architect and optimize data pipelines for batch and streaming data.
Provide technical leadership and mentorship to a team of data engineers.
Collaborate with stakeholders to define project requirements and deliverables.
Ensure best practices in data security, governance, and compliance.
Troubleshoot and resolve complex technical issues in Databricks settings.
Stay updated on the latest Databricks features and industry trends.
Key Technical Skills & Responsibilities
Experience in data engineering using Databricks or Apache Spark-based platforms.
Proven track record of building and optimizing ETL/ELT pipelines for batch and streaming data ingestion.
Hands-on experience with Azure services such as Azure Data Factory, Azure Data Lake Storage, Azure Databricks, Azure Synapse Analytics, or Azure SQL Data Warehouse.
Proficiency in programming languages such as Python, Scala, SQL for data processing and transformation.
Expertise in Spark (PySpark, Spark SQL, or Scala) and Databricks notebooks for large-scale data processing.
Familiarity with Delta Lake, Delta Live Tables, and medallion architecture for data lakehouse implementations.
Experience with orchestration tools like Azure Data Factory or Databricks Jobs for scheduling and automation.
Design and implement the Azure key vault and scoped credentials.
Knowledge of Git for source control and CI/CD integration for Databricks workflows, cost optimization, performance tuning.
Familiarity with Unity Catalog, RBAC, or enterprise-level Databricks setups.
Ability to create reusable components, templates, and documentation to standardize dat
📌 Technical Architect Pune
🏢 Eviden
📍 Pune