Experience: 6–8 Years

Location: Hyderabad

Employment Type: Contract

Open Positions: 1

Budget : Open

Duration: 1 year

Job Summary

We are seeking a highly skilled Lead Data Engineer with 6–8 years of experience in designing and building

scalable, cloud-native data platforms and high-performance data pipelines. The ideal candidate will have

expertise in developing batch and real-time data processing systems that support analytics, AI/ML

initiatives, and enterprise data solutions across domains such as Banking, FinTech, Consulting, and SaaS.

This role requires strong technical leadership, hands-on development capabilities, and the ability to

collaborate with cross-functional teams to deliver reliable and scalable data solutions.

Key Responsibilities

Data Engineering & Platform Development

 Design, develop, and maintain end-to-end data pipelines for batch and real-time processing.

 Build scalable ETL/ELT frameworks using Python and SQL.

 Implement and manage workflow orchestration using tools such as Apache Airflow.

 Design efficient data models, schemas, and transformation layers to support analytics and

downstream applications.

 Develop data ingestion pipelines from various sources, including:

o APIs

o Relational and NoSQL Databases

o Event Streams

o External Systems

Streaming & Performance Optimization

 Design and optimize real-time data streaming solutions using platforms such as Kafka.

 Enhance pipeline performance for:

o High Throughput

o Low Latency

o Cost Efficiency

o Reliability

 Implement monitoring, alerting,



and logging mechanisms to ensure platform stability and SLA

compliance.

Cloud, DevOps & Engineering Best Practices

 Develop and manage data solutions primarily on Microsoft Azure.

 Leverage AWS or GCP experience where applicable.

 Utilize Docker and CI/CD pipelines for deployment automation and version control.

 Follow software engineering best practices, including:

o Code Reviews

o Automated Testing

o Documentation Standards

o Version Control Management

 Ensure data quality, governance, validation, and compliance across all data pipelines.

AI & Advanced Data Engineering

 Support the creation of AI/ML-ready datasets and feature engineering pipelines.

 Build and maintain Feature Stores and MLOps workflows.

 Develop or integrate AI-enabled solutions, including:

o LLM-based Data Pipelines

o Document Processing Workflows

o ETL Automation

o Retrieval-Augmented Generation (RAG) Architectures

 Collaborate closely with Data Scientists, ML Engineers, and Analytics teams to enable AI-driven

business outcomes.

Leadership & Collaboration

 Translate business requirements into scalable technical solutions.

 Partner with Product, Analytics, Engineering,



and Business stakeholders.

 Mentor junior engineers and provide technical guidance.

 Participate in architecture discussions and contribute to platform design decisions.

 Drive best practices and continuous improvement initiatives across the data engineering team.

Required Skills & Experience

Must-Have Skills

 5–8+ years of experience in Data Engineering.

 Strong programming expertise in Python.

 Advanced SQL skills and experience with complex data transformations.

 Hands-on experience building ETL/ELT pipelines.

 Experience with cloud platforms:

o Microsoft Azure (Preferred)

o AWS

o GCP

 Experience with real-time data streaming technologies such as Kafka.

 Robust experience with Apache Airflow or similar orchestration tools.

 Solid understanding of:

o Data Modeling

o Data Warehousing Concepts

o Data Architecture Principles

Preferred Skills

Data Platforms & Technologies

 Snowflake

 BigQuery

 Amazon Redshift

 Delta Lake

 dbt

 Apache Spark / PySpark

AI & Machine Learning

 MLOps

 Feature Stores

 MLflow

 Large Language Models (LLMs)

 Generative AI

 Retrieval-Augmented Generation (RAG)

Domain Experience

 Banking

 FinTech

 Consulting

 Enterprise SaaS

Leadership

 Prior experience in technical leadership roles.

 Client-facing stakeholder management experience.

 Experience driving architecture and solution design discussions.

📌 Lead Data Engineer (India)
🏢 10decoders
📍 India