Data Engineer (Bengaluru)

Data Engineer (Bengaluru)

16 Nov
|
NTT DATA Services
|
Bengaluru

16 Nov

NTT DATA Services

Bengaluru

"Position Overview We are seeking an experienced Databricks Data Engineer to design and enhance enterprise document-processing and data-extraction solutions. The role will focus on ingesting structured and unstructured content, extracting business and policy information using LLM-based approaches, improving extraction accuracy, and making validated information available to downstream applications through an ontology-driven data layer. The engineer will join a small, hands-on team working on document intelligence solutions supporting brokerage and underwriting workflows. A major focus will be redesigning existing pipelines to improve accuracy, performance, scalability, and maintainability while supporting a broader migration from Microsoft Fabric to Databricks.

Key Responsibilities
• Design, build, and optimize Databricks data pipelines for document ingestion, classification, extraction, validation, and downstream consumption.
• Process information from multiple document and communication formats, including: o PDF documents o Emails and attachments o Excel/XLSX files o Insurance applications o Statements of Values (SOVs) o Other structured and unstructured documents
• Improve existing extraction pipelines where LLMs produce false positives, incorrect metadata, or inaccurate field values.
• Develop validation and quality-control approaches that improve the reliability of extracted information.
• Implement document classification and content-aware extraction strategies.
• Handle source-specific extraction rules, such as extracting particular attributes from an email rather than from attached documents.
• Design pipelines that transform extracted information into consistent, structured JSON outputs for downstream applications.




• Contribute to the design and implementation of a custom ontology layer that represents key insurance business entities and relationships.
• Map extracted information into the ontology so applications can consume standardized business concepts rather than raw document fields.
• Address performance challenges associated with processing large or complex Excel/XLSX and SOV files.
• Optimize pipeline scalability and concurrency for production workloads.
• Work with PostgreSQL/Lakebase-based persistence for final structured outputs.
• Collaborate closely with brokerage, underwriting, AI, and data engineering teams to connect extracted application data with downstream underwriting processes. Required Qualifications
• Strong hands-on experience with Databricks and production data engineering.
• Strong proficiency in Python, PySpark, and SQL.
• Experience building scalable ingestion and transformation pipelines on the Databricks platform.
• Experience processing unstructured and semi-structured data, particularly PDFs, emails, Excel files, JSON, and business documents.
• Experience integrating LLMs or AI models into data-processing pipelines.
• Understanding of prompt-based and structured data extraction from documents.
• Experience implementing data validation, reconciliation, or quality controls for AI-generated outputs.
• Solid understanding of data modeling, metadata, schemas, entity relationships, and semantic/ontology concepts.
• Experience designing APIs, JSON structures, or data products for downstream application consumption.
• Experience troubleshooting performance, scalability, and concurrency issues in production data pipelines.
• Experience working with cloud-based data architectures. Minimum 7+ years of experience in data engineering, software development, or production data pipeline development. Minimum 4+ years of hands-on experience with Databricks and scalable data-processing pipelines."

📌 Data Engineer (Bengaluru)
🏢 NTT DATA Services
📍 Bengaluru

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: data engineer (bengaluru) / bengaluru

Subscribe to this job alert:

Get the latest job offers by email for: data engineer (bengaluru) / bengaluru