Senior Data Engineer to build next-generation GenAI data platforms that power LLM-based applications. This role focuses on designing scalable pipelines for unstructured and multi-modal data and enabling RAG (Retrieval-Augmented Generation), embeddings, and AI copilots.

Responsibilities

- Own the end to end ML lifecycle, including data ingestion, feature engineering, training, evaluation, deployment, monitoring, retraining, and rollback
- Design, build, and operate production grade ML pipelines using Azure native services with solid CI/CD and automation practices
- Use Azure Machine Learning and MLflow for experiment tracking, model registry, and governed promotion across Dev/Test/Prod environments
- Design and deploy Generative AI solutions using Azure OpenAI, embeddings, vector search, and RAG pipelines
- Build Agentic AI workflows with multi step reasoning, tool usage, guardrails, observability, reliability, and cost control




- Build scalable data and feature pipelines using Azure Databricks (batch and streaming)
- Build scalable data pipelines for text, documents, logs, and multi-modal data
- Develop RAG pipelines including chunking, embedding, and retrieval workflows
- Design and manage vector search systems (Azure AI Search, Pinecone, etc.)
- Build batch + real-time data ingestion pipelines using Spark and Kafka

Required Skills

- 6–8 years of experience in Data Engineering
- Solid hands-on experience with:

o Python / PySpark / SQL o Apache Spark, Airflow, Kafka o Handling unstructured data (JSON, logs, documents, PDFs)

- Experience with cloud platforms (Azure preferred)
- Exposure to LLM / GenAI pipelines (RAG, embeddings, vector DBs)

📌 Senior Data Engineer / Gen AI (Hyderabad)
🏢 HCL INDIA
📍 Hyderabad