Data Lead - Azure Data Factory (ADF), Databricks

Pune, Maharashtra

Job Summary

We are seeking a highly skilled Snowflake Engineer with 8+ years of overall IT experience, including 3+ years of strong hands-on experience in Snowflake-based data engineering. The ideal candidate brings deep expertise in contemporary data pipeline engineering, metadata-driven ingestion, and AI-enabled data processing, along with strong consultative and stakeholder engagement skills.

Key Responsibilities

Key Responsibilities

1. Data Pipeline Engineering

- Design and develop scalable, high-performance data pipelines using SnowSQL, Snowpark, and Snowflake native capabilities
- Build and optimize ELT/ETL frameworks supporting both batch and real-time workloads
- Implement robust ingestion pipelines from diverse enterprise systems

2. Metadata-Driven Ingestion

- Design and implement metadata-driven ingestion frameworks for scalable onboarding of datasets
- Handle ingestion across multiple formats including XML, CSV, JSON, and Parquet
- Automate ingestion, validation, and transformation processes

3. AI/GenAI Integration (Cortex AI)

- Leverage Snowflake Cortex AI capabilities to enable AI-driven data transformations, enrichment, and insights generation
- Collaborate with business and analytics teams to enable AI-powered data consumption use cases
- Contribute to building intelligent, automated data pipelines and semantic layers

4. Data Modeling & Transformation

- Develop scalable data models supporting analytics and reporting layers
- Build transformations using Snowpark, SQL, and Snowflake native features
- Optimize query performance and data access patterns

5. Data Delivery & Reporting Enablement





- Deliver analytics-ready datasets in predefined formats for reporting tools (Power BI, Tableau, etc.)
- Work closely with business stakeholders to ensure data usability and alignment with reporting requirements

6. Performance Optimization & Governance

- Implement best practices for performance tuning, clustering, and cost optimization
- Ensure data quality, governance, security, and compliance standards
- Monitor, troubleshoot, and continuously improve pipelines

7. Stakeholder Engagement & Technical Consulting

- Provide technical consultation on data architecture, ingestion strategies, and Snowflake capabilities
- Act as a trusted advisor in data engineering and platform decisions

Skill Requirements

Required Skills & Experience

Skill Category

Must-Have Skill

Expected Experience / Proficiency

Priority

Overall Experience

8+ years of overall IT experience

Strong background in data engineering, enterprise data platforms, and production delivery

Must Have

Snowflake Engineering

Snowflake hands-on engineering

Minimum 3+ years of experience with Snowflake platform implementation and support

Must Have

Snowflake Core Tools

SnowSQL, Snowpark, Snowpipe, Streams, and Tasks

Ability to design, build, automate, and operationalize Snowflake-native data pipelines

Must Have

Data Pipeline Engineering

ELT/ETL pipeline design and development

Experience building scalable batch and near real-time data pipelines





Must Have

Metadata-Driven Ingestion

Metadata-driven ingestion framework design

Ability to create reusable ingestion patterns for onboarding multiple datasets efficiently

Must Have

File Formats

Parquet, CSV, XML, and JSON

Hands-on experience handling structured and semi-structured data formats; Parquet is mandatory

Must Have

SQL & Data Modeling

Advanced SQL and scalable data modeling

Strong ability to write optimized SQL, design analytical models, and support reporting layers

Must Have

Snowflake Cortex AI

Hands-on experience or strong exposure to Snowflake Cortex AI

Ability to support AI-driven data transformation, enrichment, semantic search, or insight-generation use cases

Must Have

Performance Optimization

Snowflake performance tuning and cost optimization

Experience with query optimization, clustering strategy, warehouse sizing, and monitoring

Must Have

Governance & Security

Data quality, security, governance, and compliance

Ability to apply enterprise-grade controls, access patterns, data validation, and audit readiness

Must Have

Reporting Enablement

Delivery of analytics-ready datasets

Experience preparing curated datasets for BI/reporting tools such as Power BI, Tableau, or similar platforms

Must Have

Stakeholder Management

Communication, presentation, and consulting mindset

Ability to engage with business, analytics, and technology stakeholders and provide solution guidance

Must Have

Good-to-Have Skills

- Experience with DBT (Data Build Tool)
- Knowledge of PL/SQL or procedural SQL programming
- Exposure to cloud platforms (AWS/Azure/GCP)
- Familiarity with CI/CD pipelines and version control systems

📌 Data Lead - Azure Data Factory (ADF), Databricks (India)
🏢 HCLTech
📍 India