Senior Data Engineer to build next-generation GenAI data platforms that power LLM-based applications. This role focuses on designing scalable pipelines for unstructured and multi-modal data and enabling RAG (Retrieval-Augmented Generation), embeddings, and AI copilots.
Responsibilities:
• Own the end to end ML lifecycle, including data ingestion, feature engineering, training, evaluation, deployment, monitoring, retraining, and rollback
• Design, build, and operate production grade ML pipelines using Azure native services with solid CI/CD and automation practices
• Use Azure Machine Learning and MLflow for experiment tracking, model registry, and governed promotion across Dev/Test/Prod environments
• Design and deploy Generative AI solutions using Azure OpenAI, embeddings, vector search, and RAG pipelines
• Build Agentic AI workflows with multi step reasoning, tool usage, guardrails, observability, reliability, and cost control




• Build scalable data and feature pipelines using Azure Databricks (batch and streaming)
• Build scalable data pipelines for text, documents, logs, and multi-modal data
• Develop RAG pipelines including chunking, embedding, and retrieval workflows
• Design and manage vector search systems (Azure AI Search, Pinecone, etc.)
• Build batch + real-time data ingestion pipelines using Spark and Kafka
Required Skills
• 6–8 years of experience in Data Engineering
• Robust hands-on experience with:
o Python / PySpark / SQL
o Apache Spark, Airflow, Kafka
o Handling unstructured data (JSON, logs, documents, PDFs)
• Experience with cloud platforms (Azure preferred)
• Exposure to LLM / GenAI pipelines (RAG, embeddings, vector DBs)

📌 Senior Data Engineer (Hyderabad)
🏢 HCL INDIA
📍 Hyderabad