Data Engineer (Hyderabad)

Data Engineer (Hyderabad)

17 Nov
|
Cybage Software
|
Hyderabad

17 Nov

Cybage Software

Hyderabad

Must Have :

● Strong Python experience required.

● Experience building and operating web scraping (with Scrapy) and data engineering systems in production is a plus - Preferable

● Demonstrated debugging strength: a methodical approach to reproducing, isolating, and root-causing failures in code you did not write, against sources you do not control.

● Chrome DevTools used as a reverse-engineering tool: finding the API behind the page, replaying requests,

and understanding network and auth flows - not just inspecting elements.

● Hands-on MongoDB: queries, aggregation, indexes, and ETL on large document collections.

● Strong understanding of ETL and data engineering concepts, including extraction, transformation,

validation, deduplication, incremental processing, backfills, and data-quality checks.

● Production crawl operations: scheduling, retries, rate limiting, logging, and recovery when a source breaks overnight.

● Multi-format extraction, particularly PDF, where the structure is never where you want it.

● Git and collaborative development workflows, including code review.

Good to Have :

● Browser automation for JS-rendered sources (Playwright, Selenium).

● Crawl orchestration: ScrapyKeeper, Scrapyd, or Airflow.

● Hierarchical or citation-style document modelling.

● Proxy and anti-bot handling within legal and contractual limits.

Role in brief: A hands-on Scrapy and data engineer who builds, runs and debugs production pipelines that ingest global regulatory content into MongoDB, working closely with a legal research team on data quality.

Build





- Design and own Scrapy spiders and ETL pipelines for global regulatory sources.
- Extract and normalize HTML, JSON/XML APIs, PDFs and RTF/DOC into versioned datasets, keeping document relationships, metadata and secure identifiers.
- Build item pipelines that persist to MongoDB and object storage.
- Reverse engineer hard sources (hidden APIs, JavaScript flows, tokens, anti-bot measures) using Chrome DevTools and logs.

Operate

- Run scheduled crawls through ScrapyKeeper/Scrapyd, and resolve stuck or failed jobs.
- Improve reliability with incremental and full loads, output validation, and alerts that catch silent failures.
- Use MongoDB queries to inspect, backfill and correct data safely.

Debug

- Own triage when a source breaks: reproduce, compare against the live page, and find the root cause before changing code.
- Tackle the hard bugs where a crawl succeeds but produces wrong content (truncated documents, duplicate or unstable IDs, missing versions).
- Write short postmortems and turn each into a validation check, test or fix.

Improve speed

- Build reusable base spiders and parsers so a new source takes days, not weeks.
- Own metrics such as time-to-launch, failure rate and manual intervention hours.
- Use AI-assisted development tools responsibly, and evaluate new techniques with measurable before-and-after results.

Collaborate

- Work with the legal research team on schema, coverage and content quality, and treat their QA feedback as the main correctness signal.
- Document spider designs, source quirks and runbooks, and help raise code review and testing standards.

📌 Data Engineer (Hyderabad)
🏢 Cybage Software
📍 Hyderabad

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: data engineer (hyderabad) / hyderabad

Subscribe to this job alert:

Get the latest job offers by email for: data engineer (hyderabad) / hyderabad