Lead Data Engineer

We are hiring a Lead Data Engineer to design, develop, and optimize scalable cloud data platforms and modern ETL/ELT pipelines using Microsoft Fabric, Databricks, or Snowflake.

The role involves building and maintaining data lakehouse and data warehouse solutions, developing high-performance data pipelines, enabling analytics and reporting, and contributing to data engineering best practices.

Strong hands-on experience in Py Spark, SQL, Python/Scala, and cloud data platforms is required.

What You Will Do

Following are the high-level responsibilities that you will play, but are not limited to:

Design, develop, and maintain scalable data pipelines, data models, and ETL/ELT processes. Build and optimize data lake, lakehouse, and data warehouse solutions using Microsoft Fabric, Databricks, or Snowflake. Develop data processing solutions using Py Spark, Python, Scala, and SQL. Work with large and complex datasets to improve data processing performance, reliability, and scalability. Develop and maintain batch and real-time data pipelines as required. Implement data transformation, cleansing, validation, and integration processes. Work with Delta Lake, Spark, Snowflake, Microsoft Fabric, and other modern data engineering technologies. Support business analytics and self-service reporting through Power BI and other visualization platforms. Collaborate with data scientists, analysts, BI developers, and business stakeholders to deliver reliable data solutions. Implement data quality, governance, security, and monitoring practices across data pipelines. Troubleshoot and resolve data pipeline,



performance, and production issues. Contribute to the development of coding standards, reusable frameworks, and data engineering best practices. Participate in code reviews and ensure high-quality, maintainable, and scalable solutions. Implement CI/CD practices for data engineering workflows and deployments. Stay updated with emerging data engineering technologies and contribute to continuous improvement of the data ecosystem.

Required

Qualifications

Bachelor's/Master's degree in Computer Science, Information Technology, Engineering, or a related field. 6–10 years of experience in data engineering, data platform development, or related areas. Robust hands-on experience in one or more of the following:

Microsoft Fabric – Data Factory, Lakehouse, Data Warehouse Databricks – Spark, Delta Lake, Py Spark, MLflow Snowflake – Data Warehousing, Snowpipe, Performance Optimization Power BI – Data Modeling, DAX, Report Development Strong proficiency in SQL. Strong programming skills in Python or Scala. Hands-on experience with Py Spark/Apache Spark and large-scale data processing.

Experience working with Azure, AWS, or GCP cloud data services. Strong understanding of data modeling, ETL/ELT concepts, data warehousing, and data lake/lakehouse architectures. Understanding of data governance, data quality, security, and access control.

Experience with CI/CD, Git, and Dev Ops practices** for data engineering workflows. Strong analytical and problem-solving skills. Ability to work independently and collaborate effectively with cross-functional teams.

📌 Lead Data Engineer (Hyderabad)
🏢 Thinkwise Consulting
📍 Hyderabad