Summary

We are looking for an experienced Senior Data Engineer to join our team at Quantiphi, a leader in AI-first engineering and innovative GenAI solutions. In this pivotal role, you will design and architect robust data pipelines that power advanced AI applications, specifically focusing on building high-performance ingestion engines for unstructured data, vector databases, and scalable real-time streaming architectures. You will bridge the gap between complex data engineering and up-to-date AI, ensuring data quality, lineage, and observability as we deploy cutting-edge Agentic AI solutions.
Responsibilities
Design and build batch ETL pipelines that ingest large unstructured document corpora - handling PDF, RTF, and JSON file formats, consuming pre-extracted text and metadata, applying document chunking strategies, and loading processed outputs into a vector store and analytical platforms.
Design partitioning, parallelism, and throughput strategies to sustain high-volume ingestion.
Build real-time streaming ingestion pipelines consuming document creation and update events,



processing them through chunking and embedding generation.
Build vector and indexing pipelines targeting vector store platforms (e.g., Vertex AI Vector Search, Pinecone, Weaviate, Milvus, pgvector)
Manage vector store index operations: bulk load, incremental upsert, index refresh, and consistency validation post-ingestion.
Implement metadata tagging on vector records to support filtered retrieval at query time.
Implement end-to-end lineage tracking for vector records - linking each chunk back to its source document.
Implement document-level change tracking to detect upstream updates, deletions, and re-ingestion events and trigger appropriate vector store operations (upsert, delete, re-index) without full corpus re-processing.
Tag all vector records with pipeline run metadata - ingestion timestamp, pipeline version, chunking strategy version, embedding model version, to support retrieval quality de

📌 Senior Data Engineer(15-30 days joiners only) (Bengaluru)
🏢 Quantiphi
📍 Bengaluru