Smart Bhujal was born from IIT Tirupati with a vision to make water decisions data-driven. We build AI-enabled decision-support systems for groundwater, floods, irrigation and water networks — automated, compliant data platforms for research institutes and development organisations in India and worldwide.
The role
Most of the data that matters to water decisions is public but unusable — scattered across government portals, agency websites and PDFs, in inconsistent formats and several languages. We are hiring a Senior Data Engineer to build the crawler and AI-assisted extraction pipeline that turns it into structured, provenance-tracked data. It is the most technically demanding work on the team, and everything else we build sits on top of it.
Location & engagement
Employment type: Full-time, permanent
Location: In-office · Tirupati, Andhra Pradesh
Level: Senior · 5+ years
Reports to: Technical Lead
Start: Immediate
Whom we're looking for
• B.Tech with 5+ years of experience,
hands-on in data crawling/scraping and extraction. General backend experience does not substitute.
Key responsibilities
• Design and build a discovery and crawling pipeline over authoritative seed lists.
• Implement a compliance engine — robots.txt and terms-of-use handling, conservative rate limiting, caching, identifiable user-agent, graceful back-off, off-peak scheduling and proxy rotation — so crawling is polite and low-impact.
• Build LLM-assisted extraction from HTML and PDF into structured fields, with multilingual handling, confidence scoring and a low-confidence human-review queue (uncertain data is flagged, never silently written).
• Classify data sources as official vs unofficial and capture full source metadata and provenance for every value.
• Implement staged document fetching with metadata extraction and version history.
• Work to disciplined version-control, CI/CD, branching and cod