Senior Data Engineer to build next-generation GenAI data platforms that power LLM-based applications. This role focuses on designing scalable pipelines for unstructured and multi-modal data and enabling RAG (Retrieval-Augmented Generation), embeddings, and AI copilots.

Responsibilities
Own the end to end ML lifecycle, including data ingestion, feature engineering, training, evaluation, deployment, monitoring, retraining, and rollback
Design, build, and operate production grade ML pipelines using Azure native services with solid CI/CD and automation practices
Use Azure Machine Learning and MLflow for experiment tracking, model registry, and governed promotion across Dev/Test/Prod settings
Design and deploy Generative AI solutions using Azure OpenAI, embeddings, vector search, and RAG pipelines
Build Agentic AI workflows with multi step reasoning, tool usage, guardrails, observability, reliability, and cost control




Build scalable data and feature pipelines using Azure Databricks (batch and streaming)
Build scalable data pipelines for text, documents, logs, and multi-modal data
Develop RAG pipelines including chunking, embedding, and retrieval workflows
Design and manage vector search systems (Azure AI Search, Pinecone, etc.)
Build batch + real-time data ingestion pipelines using Spark and Kafka

Required Skills
6–8 years of experience in Data Engineering
Solid hands-on experience with:

o Python / PySpark / SQL o Apache Spark, Airflow, Kafka o Handling unstructured data (JSON, logs, documents, PDFs)
Experience with cloud platforms (Azure preferred)
Exposure to LLM / GenAI pipelines (RAG, embeddings, vector DBs)

📌 Senior Data Engineer / Gen Ai Hyderabad
🏢 HCL INDIA
📍 Hyderabad