Senior Data Engineer - GenAI & Unstructured Data Pipelines

Job Type: Full-time
Experience: 6-8 Years
Location: Offshore (Remote)

Job Summary

This role is ideal for professionals passionate about modern data engineering, cloud-native architectures, and Generative AI technologies.

Key Responsibilities
- Design, build, and maintain scalable data pipelines for structured, unstructured, and multi-modal data.
- Own and support the end-to-end machine learning lifecycle, including data ingestion, feature engineering, model training, evaluation, deployment, monitoring, retraining, and rollback.
- Develop production-grade ML pipelines using Azure native services with CI/CD automation and best practices.
- Utilize Azure Machine Learning and MLflow for experiment tracking, model registry, and governed deployments across environments.
- Design and implement Generative AI solutions using Azure OpenAI, embeddings, vector search, and Retrieval-Augmented Generation (RAG).
- Build Agentic AI workflows with multi-step reasoning, tool integration, observability, guardrails, reliability,



and cost optimization.
- Develop scalable batch and streaming data pipelines using Azure Databricks, Apache Spark, and Kafka.
- Create efficient RAG pipelines, including document chunking, embedding generation, indexing, and retrieval workflows.
- Design and manage vector databases and search solutions such as Azure AI Search and Pinecone.
- Build high-performance batch and real-time data ingestion pipelines using Spark and Kafka.

Required Skills & Qualifications
- 6-8 years of experience in Data Engineering.
- Robust hands-on expertise with:
- Python
- PySpark
- SQL
- Apache Spark
- Apache Airflow
- Apache Kafka
- Experience processing and managing unstructured data, including JSON, logs, documents, and PDFs.
- Strong experience with cloud platforms, preferably Microsoft Azure.
- Hands-on exposure to Generative AI technologies, including:
- Large Language Models (LLMs)
- Retrieval-Augmented Genera

📌 Ai Data Engineer (Noida)
🏢 WebSenor
📍 Noida