17 Nov
|
Cybage Software
|
Hyderabad
17 Nov
Cybage Software
Hyderabad
Must Have :
● Strong Python experience required.
● Experience building and operating web scraping (with Scrapy) and data engineering systems in production is a plus - Preferable
● Demonstrated debugging strength: a methodical approach to reproducing, isolating, and root-causing failures in code you did not write, against sources you do not control.
● Chrome DevTools used as a reverse-engineering tool: finding the API behind the page, replaying requests,
and understanding network and auth flows - not just inspecting elements.
● Hands-on MongoDB: queries, aggregation, indexes, and ETL on large document collections.
● Strong understanding of ETL and data engineering concepts, including extraction, transformation,
validation, deduplication, incremental processing, backfills, and data-quality checks.
● Production crawl operations: scheduling, retries, rate limiting, logging, and recovery when a source breaks overnight.
● Multi-format extraction, particularly PDF, where the structure is never where you want it.
● Git and collaborative development workflows, including code review.
Good to Have :
● Browser automation for JS-rendered sources (Playwright, Selenium).
● Crawl orchestration: ScrapyKeeper, Scrapyd, or Airflow.
● Hierarchical or citation-style document modelling.
● Proxy and anti-bot handling within legal and contractual limits.
Role in brief: A hands-on Scrapy and data engineer who builds, runs and debugs production pipelines that ingest global regulatory content into MongoDB, working closely with a legal research team on data quality.
Build
- Design and own Scrapy spiders and ETL pipelines for global regulatory sources.
- Extract and normalize HTML, JSON/XML APIs, PDFs and RTF/DOC into versioned datasets, keeping document relationships, metadata and secure identifiers.
- Build item pipelines that persist to MongoDB and object storage.
- Reverse engineer hard sources (hidden APIs, JavaScript flows, tokens, anti-bot measures) using Chrome DevTools and logs.
Operate
- Run scheduled crawls through ScrapyKeeper/Scrapyd, and resolve stuck or failed jobs.
- Improve reliability with incremental and full loads, output validation, and alerts that catch silent failures.
- Use MongoDB queries to inspect, backfill and correct data safely.
Debug
- Own triage when a source breaks: reproduce, compare against the live page, and find the root cause before changing code.
- Tackle the hard bugs where a crawl succeeds but produces wrong content (truncated documents, duplicate or unstable IDs, missing versions).
- Write short postmortems and turn each into a validation check, test or fix.
Improve speed
- Build reusable base spiders and parsers so a new source takes days, not weeks.
- Own metrics such as time-to-launch, failure rate and manual intervention hours.
- Use AI-assisted development tools responsibly, and evaluate new techniques with measurable before-and-after results.
Collaborate
- Work with the legal research team on schema, coverage and content quality, and treat their QA feedback as the main correctness signal.
- Document spider designs, source quirks and runbooks, and help raise code review and testing standards.
📌 Data Engineer (Hyderabad)
🏢 Cybage Software
📍 Hyderabad