17 Nov
|
Cybage Software
|
Hyderabad
17 Nov
Cybage Software
Hyderabad
Job Description
Must Have : ● Strong Python experience required. ● Experience building and operating web scraping (with Scrapy) and data engineering systems in production is a plus
- Preferable ● Demonstrated debugging strength: a methodical approach to reproducing, isolating, and root-causing failures in code you did not write, against sources you do not control. ● Chrome DevTools used as a reverse-engineering tool: finding the API behind the page, replaying requests, and understanding network and auth flows
- not just inspecting elements. ● Hands-on MongoDB: queries, aggregation, indexes, and ETL on large document collections. ● Strong understanding of ETL and data engineering concepts, including extraction, transformation, validation, deduplication, incremental processing, backfills, and data-quality checks. ● Production crawl operations: scheduling, retries, rate limiting, logging, and recovery when a source breaks overnight. ● Multi-format extraction, particularly PDF, where the structure is never where you want it. ● Git and collaborative development workflows, including code review. Good to Have : ● Browser automation for JS-rendered sources (Playwright, Selenium). ● Crawl orchestration: ScrapyKeeper, Scrapyd, or Airflow. ● Hierarchical or citation-style document modelling. ● Proxy and anti-bot handling within legal and contractual limits. Role in brief: A hands-on Scrapy and data engineer who builds, runs and debugs production pipelines that ingest global regulatory content into MongoDB, working closely with a legal research team on data quality. Build
• Design and own Scrapy spiders and ETL pipelines for global regulatory sources.
• Extract and normalize HTML, JSON/XML APIs, PDFs and RTF/DOC into versioned datasets, keeping document relationships, metadata and stable identifiers.
• Build item pipelines that persist to MongoDB and object storage.
• Reverse engineer hard sources (hidden APIs, JavaScript flows, tokens, anti-bot measures) using Chrome DevTools and logs. Operate
• Run scheduled crawls through ScrapyKeeper/Scrapyd, and resolve stuck or failed jobs.
• Improve reliability with incremental and full loads, output validation, and alerts that catch silent failures.
• Use MongoDB queries to inspect, backfill and correct data safely. Debug
• Own triage when a source breaks: reproduce, compare against the live page, and find the root cause before changing code.
• Tackle the hard bugs where a crawl succeeds but produces wrong content (truncated documents, duplicate or unstable IDs, missing versions).
• Write short postmortems and turn each into a validation check, test or fix. Improve speed
• Build reusable base spiders and parsers so a recent source takes days, not weeks.
• Own metrics such as time-to-launch, failure rate and manual intervention hours.
• Use AI-assisted development tools responsibly, and evaluate new techniques with measurable before-and-after results. Collaborate
• Work with the legal research team on schema, coverage and content quality, and treat their QA feedback as the main correctness signal.
• Document spider designs, source quirks and runbooks, and help raise code review and testing standards.
📌 Data Engineer (Hyderabad)
🏢 Cybage Software
📍 Hyderabad