We are seeking a highly skilled Technical Lead with robust expertise in Databricks, Python, and PySpark to lead data engineering initiatives. The ideal candidate will drive the design, development, and optimization of scalable data pipelines while mentoring a team of engineers and collaborating with cross-functional stakeholders.
Key Responsibilities
Lead the design and development of data pipelines and ETL/ELT workflows using Databricks and PySpark
Architect and implement scalable, high-performance data solutions on cloud platforms (AWS/GCP)
Collaborate with data architects, analysts, and business teams to translate requirements into technical solutions
Optimize data processing jobs for performance, reliability, and cost efficiency
Ensure data quality, governance, and security standards are followed
Mentor and guide junior engineers; perform code reviews and enforce best practices
Drive adoption of CI/CD, DevOps, and automated testing in data engineering workflows
Troubleshoot and resolve production issues, ensuring high availability of data systems
Required Skills & Qualifications




Strong experience in Python and PySpark development
Hands-on expertise with Databricks (workflows, Delta Lake, notebooks, cluster management)
Solid understanding of data engineering concepts, distributed computing, and big data processing
Experience with SQL and relational/NoSQL databases
Expertise in data modeling, partitioning, and performance tuning
Proficiency with cloud platforms (AWS/GCP equivalents)
Familiarity with Delta Lake, streaming (Structured Streaming), and batch workloads
Strong knowledge of Git, CI/CD pipelines, and DevOps practices
Experience with workflow orchestration tools (Airflow, Temporal, etc.)
Preferred Qualifications
Experience with data warehousing and lakehouse architecture
Knowledge of ML pipelines or MLOps integration
Exposure to data governance tools and frameworks
Certification in Databricks is a plus
Leadership & Soft Skills
Proven experience in technica

📌 Technical Lead – Databricks (Hyderabad)
🏢 Crisil
📍 Hyderabad