Job Description: DataBricks – Technical Lead (Exp: 8 Yrs to 12 Yrs)
We are seeking a talented and motivated DataBricks expert to join our team as a Technical Lead.
Required Skills & Qualifications
• 8 to 12 years of IT experience, with 3+ years focused on big data and Databricks.
• Strong hands-on experience with Databricks, PySpark, and Spark SQL.
• Experience with modern cloud data lakes (Azure ADLS, AWS S3).
• Hands-on experience with Azure Data Factory (ADF) or AWS Glue for orchestration.
• Robust experience with relational databases and SQL.
• Robust background in Agile delivery, CI/CD pipelines, and version control (Git).
• Experience with Delta Lake, Unity Catalog, and the medallion architecture.
• Experience with streaming data (Structured Streaming, Kafka, Event Hubs).
• Experience with Infrastructure-as-Code (Terraform).
• Experience in agile development processes using Jira and Confluence.
• Design scalable data engineering solutions and lead their delivery on Databricks.
• Understanding of the SDLC and Agile methodologies.
• Communication with the customer and producing the daily status report.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Databricks Certified Data Engineer Professional certification.
• Microsoft Certified: Fabric Data Engineer Associate (DP-700) or AWS Certified Data Engineer - Associate.
• Should have positive oral and written communication.
• Should be able to lead and mentor a team.
• Should be proactive and adaptive.
Key Responsibilities
• Design scalable, high-performance data pipelines and Lakehouse workflows using Databricks and PySpark / Spark SQL.
• Build complex ETL/ELT pipelines on cloud platforms such as Azure or AWS.
• Guide development teams, perform code reviews, and enforce engineering best practices.
• Mentor engineers on Databricks, Spark, and data engineering design patterns.
• Work with product owners, business analysts, and architects to translate requirements into technica