Job Title: Senior Data Engineer – GenAI & Unstructured Data Pipelines
Experience: 6–8 Years
Employment Type: Full-Time

Role Summary

We are seeking an experienced Senior Data Engineer to design and build next-generation GenAI-powered data platforms that enable Large Language Model (LLM) applications. The ideal candidate will have extensive expertise in developing scalable data pipelines for unstructured and multi-modal data, implementing Retrieval-Augmented Generation (RAG) architectures, embeddings, vector search, and AI copilot solutions. The role requires strong experience in Azure cloud technologies, modern data engineering frameworks, and production-grade machine learning pipelines.

Key Responsibilities

- Own the complete machine learning lifecycle including:
- Data ingestion
- Feature engineering
- Model training
- Evaluation
- Deployment
- Monitoring
- Retraining
- Rollback strategies
- Design, develop, and maintain production-grade ML pipelines using Azure-native services with robust CI/CD automation.
- Utilize Azure Machine Learning and MLflow for experiment tracking, model registry, and governed model promotion across Dev, Test, and Production environments.
- Design and implement Generative AI solutions leveraging:
- Azure OpenAI
- Embeddings
- Vector Search
- Retrieval-Augmented Generation (RAG)
- Build Agentic AI workflows with:
- Multi-step reasoning
- Tool integrations
- Guardrails
- Observability
- Reliability
- Cost optimization
- Develop scalable batch and streaming data pipelines using Azure Databricks.
- Build scalable pipelines for processing:
- Text
- Documents
- Logs
- PDFs
- Multi-modal datasets
- Design and implement RAG pipelines including:
- Data chunking
- Embedding generation
- Retrieval workflows
- Design, optimize, and manage vector search platforms such as Azure AI Search and Pinecone.




- Develop both batch and real-time data ingestion pipelines using Apache Spark and Kafka.

Required Technical SkillsProgramming

- Python
- PySpark
- SQL

Big Data Technologies

- Apache Spark
- Apache Airflow
- Apache Kafka

Cloud Platforms

- Microsoft Azure (Preferred)

Azure Services

- Azure Databricks
- Azure Machine Learning
- Azure OpenAI
- Azure AI Search

Machine Learning & MLOps

- MLflow
- CI/CD for ML pipelines
- Model Registry
- Model Monitoring
- Experiment Tracking

Generative AI

- Large Language Models (LLMs)
- Retrieval-Augmented Generation (RAG)
- Embeddings
- Vector Databases
- AI Copilots
- Agentic AI Workflows

Data Engineering

- Batch Processing
- Streaming Pipelines
- Feature Engineering
- Unstructured Data Processing

Data Types

- JSON
- Logs
- Documents
- PDFs
- Multi-modal Data

Required Experience

- 6–8 years of hands-on experience in Data Engineering.
- Strong expertise in Python, PySpark, and SQL.
- Experience with Apache Spark, Airflow, and Kafka.
- Hands-on experience processing unstructured data such as JSON, logs, documents, and PDFs.
- Experience working on Microsoft Azure cloud platforms.
- Exposure to GenAI technologies including RAG pipelines, embeddings, and vector databases.

Preferred Qualifications

- Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.
- Azure certifications in Data Engineering or AI are an advantage.
- Experience designing enterprise-scale AI and analytics platforms.
- Familiarity with MLOps best practices and production AI deployments.

Soft Skills

- Excellent analytical and problem-solving abilities.
- Solid communication and stakeholder management skills.
- Ability to work independently and within cross-functional teams.
- Strong ownership mindset with attention to quality and scalability.
- Passion for emerging AI and cloud technologies.

Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)

📌 Senior Data Engineer – GenAI & Unstructured Data Pipelines (India)
🏢 WebSenor
📍 India