Company Description

NIQ is the world’s leading consumer intelligence company, delivering the most complete understanding of consumer buying behavior and revealing new pathways to growth. In 2023, NIQ combined with GfK, bringing together the two industry leaders with unparalleled global reach. With a holistic retail read and the most comprehensive consumer insights—delivered with advanced analytics through state-of-the-art platforms—NIQ delivers the Full View. NIQ is an Advent International portfolio company with operations in 100+ markets, covering more than 90% of the world’s population.

You will be responsible for designing flexible and scalable data pipelines, enabling retrieval systems, and ensuring high-quality data flows across the GenAI lifecycle—from experimentation to production deployment.

Core Responsibilities

Data Engineering & Pipeline Development

- Design and build scalable batch and near real-time data pipelines using contemporary data processing frameworks.
- Develop robust ETL/ELT workflows for ingesting and transforming structured and unstructured data (documents, PDFs, APIs, logs, etc.).
- Leverage cloud-native orchestration and data processing solutions to ensure reliability and scalability.
- Implement reusable data frameworks that support rapid experimentation and iteration cycles .
- Ensure data quality through validation, schema enforcement, and automated checks.

GenAI Data Preparation & Experimentation

- Prepare and curate datasets for GenAI use cases including RAG, embeddings, and fine-tuning workflows .
- Implement data processing steps like chunking, tokenization, metadata enrichment, and semantic structuring .
- Enable fast experimentation loops by supporting dynamic datasets and evaluation pipelines.




- Collaborate with engineers and product teams to iterate quickly on features and experiments.
- Transition experimental pipelines into production-ready, robust workflows .

Vector Databases & Retrieval Systems

- Build and maintain embedding pipelines using LLM providers and open-source models.
- Design and optimize retrieval systems using cloud-native vector databases and hybrid storage solutions .
- Work with relational databases supporting vector capabilities (e.g., PostgreSQL with vector extensions ).
- Implement and optimize RAG pipelines , including indexing, retrieval, ranking, and refresh strategies.
- Manage lifecycle of embeddings, vector indexes, and retrieval datasets.

Data Storage & Platform Engineering

- Work with cloud-native data platforms and storage solutions (data lakes, lakehouses, object storage).
- Design efficient storage schemas for both analytical and retrieval workloads.
- Optimize relational and hybrid data stores for low-latency, high-throughput access patterns.
- Ensure cost-effective and scalable data storage strategies.

Productionization & Scalability

- Convert experimental workflows into scalable, reliable production pipelines .
- Optimize pipelines for performance, cost, and reliability.
- Implement incremental processing, caching, and efficient refresh strategies.

Monitoring & Data Observability

- Implement monitoring for pipeline health, data freshness, and quality.




- Track dataset drift, embedding drift, and retrieval effectiveness.
- Build logging, alerting, and observability frameworks for data systems.

Collaboration & Cross-Functional Work

- Partner closely with engineering teams, product leadership, and data scientists to define and deliver data solutions.
- Act as a bridge between rapid experimentation and production engineering .
- Contribute to architecture decisions and GenAI data best practices.
- Document pipelines, architectures, and data models clearly.

Nice-to-Have / Growth Areas

- Experience with GenAI frameworks (e.g., LangChain, LlamaIndex, or similar)

- Exposure to knowledge graphs and graph-based retrieval approaches :

- Understanding of data governance, lineage, and cataloging

- Experiment tracking and dataset versioning

Experience working with multi-modal datasets (text, image, audio) Qualifications

- 2-3 years of experience in Data Engineering or related roles.
- Strong proficiency in Python and SQL .
- Hands-on experience with modern data processing frameworks and orchestration tools .
- Experience working with cloud-native data platforms (Azure, AWS, or GCP).
- Experience with relational databases such as PostgreSQL , including extensions for advanced workloads (e.g., vector storage).
- Strong understanding of building scalable data pipelines for both structured and unstructured data.
- Familiarity with GenAI concepts such as LLMs, embeddings, and RAG architectures .

Soft Skills

- Strong collaborator comfortable working with engineers, product managers, and leadership .
- Ability to balance rapid experimentation with production rigor .
- Strong problem-solving and debugging capabilities acro

📌 Gen Ai Data Engineer (Pune)
🏢 Nielseniq
📍 Pune