(JD)

Lead / Senior Data Engineer

Location: Hyderabad, India

Experience: 8+ Years

Employment Type: Full-Time

Notice Period : Immediate / 15 Days.

About the Role

We are looking for a highly skilled and results-driven Lead / Senior Data Engineer with extensive experience in designing, developing, and managing enterprise-scale data platforms across Azure and AWS cloud ecosystems. The ideal candidate will bring strong expertise in Azure Databricks, Snowflake, Delta Lake, Unity Catalog, Delta Live Tables (DLT), Azure Data Factory, Python, Py Spark, and SQL , combined with proven leadership experience in delivering modern data warehouse and lakehouse solutions.

This role requires a hands-on technical leader who can drive architecture decisions, build scalable and governed data platforms, mentor engineering teams, and collaborate closely with business stakeholders to deliver data-driven solutions that enable analytics, reporting, and AI/ML initiatives.

Key Responsibilities

Data Architecture &

- Engineering

Design, develop, and maintain scalable and high-performance ETL/ELT pipelines using Azure Data Factory, Azure Databricks, Python, Py Spark, and SQL.

Build metadata-driven and parameterized data ingestion frameworks supporting both batch and real-time processing.

Architect modern Lakehouse and Data Warehouse solutions leveraging Azure and AWS cloud technologies.

Develop analytics-ready datasets and enterprise data products for business intelligence, reporting, and advanced analytics use cases.

Implement robust data quality frameworks, reconciliation processes, and validation controls.

Azure Databricks &

- Lakehouse Development

Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.

Build and manage Delta Live Tables (DLT) pipelines for reliable and quality-enforced data processing.

Develop Py Spark and Spark SQL transformations for large-scale data workloads.

Implement Unity Catalog governance, including data lineage, auditing, metadata management, and fine-grained access controls.

Optimize Databricks workloads through partitioning strategies, caching, broadcast joins, Adaptive Query Execution (AQE),



and data-skew handling.

Snowflake Data Warehousing

Design and implement Snowflake-based data warehouse solutions.

Develop and optimize Snowflake schemas, data models, RBAC frameworks, and data ingestion processes using Snowpipe and COPY operations.

Perform SQL query tuning, warehouse sizing, performance optimization, and cost management.

Support reporting, analytics, and enterprise BI workloads through scalable and efficient data models.

Cloud &

- Streaming Solutions

Develop real-time and event-driven data processing solutions using Azure Event Hubs, AWS Kinesis, Azure Functions, and AWS Lambda.

Build scalable ingestion frameworks for streaming and batch data sources.

Support hybrid and multi-cloud data ecosystems across Azure and AWS platforms.

Data Modeling &

- Governance

Design dimensional data models using Star Schema and Snowflake Schema methodologies.

Implement Slowly Changing Dimensions (SCD Type 1, 2, and 3).

Support Data Mesh and domain-oriented data product initiatives.

Establish enterprise data governance practices, security controls, and data stewardship standards.

Dev Ops &

- Automation

Develop and maintain CI/CD pipelines using Azure Dev Ops and Git Hub Actions.

Implement automated deployment, monitoring, testing, and release management processes.

Promote engineering best practices through code reviews, version control standards, and development frameworks.

Leadership &

- Stakeholder Management

Lead technical design discussions, architectural reviews, and project delivery activities.

Mentor and guide data engineering teams through technical coaching and knowledge sharing.

Collaborate with business stakeholders, architects, data scientists, and analytics teams to define and implement data solutions.





Drive technical strategy and ensure successful project execution from design through production support.

Required Qualifications

Education

Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related discipline.

Experience

8+ years of experience in Data Engineering, Data Warehousing, and Cloud Data Platforms.

Proven experience leading enterprise data transformation and modernization initiatives.

Solid track record of delivering scalable data solutions in Azure cloud environments.

Technical Skills

Cloud Platforms

Microsoft Azure

Amazon Web Services (AWS)

Azure Technologies

Azure Databricks

Azure Data Factory (ADF)

Azure Synapse Analytics

Azure Data Lake Storage Gen2 (ADLS Gen2)

Azure SQL Database

Azure Event Hubs

Azure Functions

Microsoft Fabric

Databricks &

- Data Governance

Py Spark

Spark SQL

Delta Lake

Delta Live Tables (DLT)

Unity Catalog

Databricks Workflows

Data Warehousing

Snowflake

Azure Synapse Analytics

Amazon Redshift

Programming Languages

Python

Py Spark

SQL / T-SQL

Pandas

AWS Services

AWS Glue

Amazon S3

Amazon Athena

AWS Lambda

AWS Step Functions

AWS IAM

Amazon Cloud Watch

Data Modeling

Medallion Architecture

Star Schema &

- Snowflake Schema

Data Warehousing

Data Mesh

Lakehouse Architecture

SCD Type 1, Type 2, and Type 3

Dev Ops &

- Collaboration

Azure Dev Ops

Git Hub Actions

Git Version Control

Agile/Scrum Methodologies

Preferred Qualifications

Databricks Certification (Associate/Professional).

Snowflake Certification.

Microsoft Azure Data Engineering Certifications.

Experience supporting AI/ML and advanced analytics workloads.

Experience implementing enterprise data governance frameworks.

Exposure to Microsoft Fabric and modern Lakehouse platforms.

Key Competencies

Technical Leadership

Solution Architecture

Data Engineering Best Practices

Performance Optimization

Problem Solving &

- Root Cause Analysis

Stakeholder Management

Team Mentoring &

- Coaching

Strategic Decision Making

Communication &

- Collaboration

📌 Senior Data Engineer (Hyderabad)
🏢 Anblicks
📍 Hyderabad