– Data Engineer (Databricks | Snowflake | Python)

Position: Senior Data Engineer – Databricks, Snowflake & Python

Experience: 5+ Years

Location: Hyderabad, India

Employment Type: Full-Time

About the Role

We are seeking a highly skilled Data Engineer with strong expertise in Databricks, Snowflake, and Python to design, build, and optimize scalable data platforms and modern data pipelines. The ideal candidate should have hands-on experience working with cloud-based data engineering solutions, data lakes, ETL/ELT frameworks, and large-scale analytics platforms.

This role requires close collaboration with data architects, analysts, AI/ML engineers, and business stakeholders to deliver reliable, high-performance data solutions.

Key Responsibilities

= Design and develop scalable data pipelines using Databricks and Python.

= Build and optimize ETL/ELT workflows for structured and semi-structured data.

= Develop and maintain data models in Snowflake.

= Implement Delta Lake architecture and data lakehouse solutions.

= Create high-performance Spark applications using PySpark.

= Develop reusable data ingestion frameworks.

= Optimize Snowflake warehouses for performance and cost.

= Integrate multiple data sources including APIs, databases, files, and streaming platforms.

= Implement data quality, validation, and monitoring frameworks.

= Build orchestration workflows using Azure Data Factory, Airflow, or similar tools.

= Collaborate with BI teams to support reporting and analytics.

= Implement CI/CD pipelines for data engineering projects.

= Ensure security, governance, and compliance of enterprise data.

= Troubleshoot production issues and optimize existing data pipelines.

= Mentor junior engineers and participate in design reviews.

Required Skills

Databricks





= Azure Databricks / AWS Databricks

= Delta Lake

= Unity Catalog

= Databricks Workflows

= Databricks SQL

= Delta Live Tables (DLT)

= MLflow (preferred)

= Lakehouse Architecture

Snowflake

= Snowflake Architecture

= Virtual Warehouses

= Data Sharing

= Snowpipe

= Streams & Tasks

= Time Travel

= Cloning

= Snowpark

= Query Optimization

= Performance Tuning

= Data Modeling

= RBAC Security

Python

= Advanced Python Programming

= PySpark

= Pandas

= NumPy

= REST API Integration

= Object-Oriented Programming

= Logging & Exception Handling

= Unit Testing

= Packaging & Modular Development

SQL

= Advanced SQL

= Complex Joins

= Window Functions

= CTEs

= Stored Procedures

= Performance Optimization

Cloud Platforms (One or More)

= Microsoft Azure

= AWS

= Google Cloud Platform

Data Engineering

= ETL / ELT

= Data Warehousing

= Data Lake

= Lakehouse

= Data Modeling

= Batch Processing

= Streaming Data

= Metadata Management

= Data Governance

DevOps

= Git

= Azure DevOps

= Jenkins

= GitHub Actions

= CI/CD Pipelines

= Docker (preferred)

Preferred Technologies

= Azure Data Factory

= Apache Airflow

= Kafka

= Event Hub

= Terraform

= Kubernetes

= Power BI

= Tableau

= dbt

= Excellent Expectations

= Microsoft Fabric (preferred)

Qualifications

= Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.





= 5+ years of experience in Data Engineering.

= At least 3 years of hands-on experience with Databricks.

= At least 3 years of experience with Snowflake.

= Strong Python development experience.

= Excellent SQL skills.

= Experience with cloud-native data platforms.

= Knowledge of data security and governance best practices.

Nice to Have

= Microsoft Fabric

= Azure Synapse Analytics

= AI/ML pipelines

= Generative AI data preparation

= LangChain

= Vector Databases

= OpenAI APIs

= Apache Iceberg

= Apache Hudi

Soft Skills

= Excellent communication skills

= Strong analytical and problem-solving abilities

= Ability to work independently

= Stakeholder management

= Team collaboration

= Agile/Scrum experience

= Documentation skills

Success Metrics

= High-performance, scalable data pipelines

= Optimized Snowflake compute costs

= Reliable and automated ETL workflows

= High data quality and governance standards

= Timely delivery of data engineering projects

= Strong collaboration with analytics and AI teams

Preferred Candidate Profile

= Hands-on experience with modern cloud data platforms.

= Strong understanding of Lakehouse architecture and enterprise data management.

= Experience supporting AI/ML workloads using Databricks and Snowflake.

= Ability to design scalable, secure, and cost-efficient data solutions.

= Exposure to healthcare, life sciences, retail, or financial services domains is an advantage.

Expected Certification

= Databricks: Databricks Certified Data Engineer Associate (minimum), Professional preferred.

= Snowflake: SnowPro Core (minimum), SnowPro Advanced: Data Engineer preferred.

= Cloud: Azure DP-203 or AWS Data Engineer – Associate

📌 Data Engineer (Hyderabad)
🏢 Avira Digital
📍 Hyderabad