Senior Data Lead - Azure Data Factory (ADF), Databricks

Experience: 7 to Not Available years

Location: Noida, India

Skills: Azure Databricks, Snowflake, Python, Spark, SQL, ETL, ELT, Azure Functions, Service Bus, Apache Spark, PySpark, Spark SQL, API integrations, REST, GraphQL, Terraform, CI/CD, GitHub Actions, Git, Artifactory, Sonar, Agile, Scrum

Job Summary
Seeking a Senior Data Engineer with strong expertise in cloud-based data engineering, large-scale data processing, and API-driven integrations. The ideal candidate will design, build, and optimize scalable, secure, and high‑performance data platforms to support analytics and business intelligence use cases. This role requires hands-on experience with Azure Databricks, Snowflake, Python, Spark, SQL, and modern DevOps practices, along with the ability to collaborate closely with cross‑functional teams to deliver reliable data solutions.

Key Responsibilities
Design, develop, and maintain end-to-end data pipelines using ETL/ELT patterns to ingest, transform, and deliver high‑quality data.
Architect and implement cloud-based data solutions leveraging Azure Databricks (Unity Catalog), Snowflake, Azure Functions, and Service Bus.
Build and optimize distributed data processing workflows using Apache Spark (PySpark, Spark SQL).
Develop and manage API integrations (REST, GraphQL) for data ingestion and system interoperability.
Design complex data models and schemas, ensuring scalability, data integrity, and optimal query performance.




Perform SQL performance tuning and optimization for large datasets and analytical workloads.
Implement data architecture best practices, including security, governance, and access controls.
Automate infrastructure provisioning and configuration using Terraform.
Implement and maintain CI/CD pipelines using GitHub Actions, Git, Artifactory, and Sonar.
Collaborate with product owners, architects, and analytics teams to translate business requirements into technical solutions.
Ensure high standards of data quality, reliability, and operational stability across environments.

Skill Requirements
7+ years of experience in Data Engineering, Data Integration, Data Modeling, Data Architecture, and ETL/ELT processes.
5+ years of strong hands-on experience with Python for data engineering workloads.
3+ years of experience in API design and lifecycle management (REST, GraphQL).
2+ years of experience with Apache Spark using PySpark and Spark SQL.
2+ years of experience building and deploying cloud-based solutions using: Azure Databricks (with Unity Catalog), Snowflake, Azure Functions, Azure Service Bus.
2+ years of robust SQL experience, including complex schema design and query performance optimization.
Experience with Infrastructure as Code (IaC) using Terraform.
Strong understanding of CI/CD processes and tools such as GitHub Actions, Git, Artifactory, and Sonar.
Experience working in Agile/Scrum environments with a focus on quality and automation.

Other Requirements
Relevant certifications in Snowflake, Azure Data Factory (ADF), DataBricks are a plus.

📌 Senior Data Lead Azure Data Factory (ADF), Databricks (Noida)
🏢 HCL Technologies
📍 Noida