Summary

We are looking for an experienced Senior Data Engineer to join our team at Quantiphi, a leader in AI-first engineering and creative GenAI solutions. In this pivotal role, you will design and architect robust data pipelines that power advanced AI applications, specifically focusing on building high-performance ingestion engines for unstructured data, vector databases, and scalable real-time streaming architectures. You will bridge the gap between complex data engineering and modern AI, ensuring data quality, lineage, and observability as we deploy cutting-edge Agentic AI solutions.

Responsibilities
- Design and build batch ETL pipelines that ingest large unstructured document corpora - handling PDF, RTF, and JSON file formats, consuming pre-extracted text and metadata, applying document chunking strategies, and loading processed outputs into a vector store and analytical platforms.
- Design partitioning, parallelism, and throughput strategies to sustain high-volume ingestion.
- Build real-time streaming ingestion pipelines consuming document creation and update events,



processing them through chunking and embedding generation.
- Build vector and indexing pipelines targeting vector store platforms (e.g., Vertex AI Vector Search, Pinecone, Weaviate, Milvus, pgvector)
- Manage vector store index operations: bulk load, incremental upsert, index refresh, and consistency validation post-ingestion.
- Implement metadata tagging on vector records to support filtered retrieval at query time.
- Implement end-to-end lineage tracking for vector records - linking each chunk back to its source document.
- Implement document-level change tracking to detect upstream updates, deletions, and re-ingestion events and trigger appropriate vector store operations (upsert, delete, re-index) without full corpus re-processing.
- Tag all vector records with pipeline run metadata - ingestion timestamp, pipeline version, chunking strategy version, embedding model version, to support r

📌 Senior Data Engineer(15-30 days joiners only) (Bangalore Metropolitan Area)
🏢 Quantiphi
📍 Bangalore Metropolitan Area