Data Engineer (35+ years)

Responsibilities

- Build connectors: enterprise DBs, APIs, CSV/Excel, SFTP, ERP, GIS, SCADA, meter-data, IoT

- Develop batch and incremental ingestion pipelines; CDC and streaming pipelines

- Create reusable source-to-canonical mapping configurations

- Normalize inconsistent identifiers, timestamps, units, entity names

- Implement entity resolution, deduplication, master-data reconciliation

- Build pipelines loading data into PostgreSQL, Apache Jena/Fuseki, Apache AGE/Neo4j, OpenSearch, object storage

- Support disconnected/air-gapped ingestion from files and internal systems

- Build automated data-quality rules (schema conformity, missing values, duplicates, invalid IDs, unit inconsistencies, timestamp gaps, outliers, referential integrity, freshness, reconciliation)

- Implement quality scoring at dataset/record/attribute level

- Integrate validation with SHACL-based semantic validation

- Quarantine invalid data, produce validation reports

- Track lineage from source records to findings/model outputs

- Build graph-ready datasets and graph-loading pipelines

Core Skills

- Strong Python, advanced SQL/PostgreSQL, pandas/Polars, NumPy,



Apache Arrow

- Apache Airflow, NiFi, Kafka/Redpanda, Debezium, Spark, Trino, Iceberg, Parquet/Avro

- MinIO/S3-compatible storage

- Schema evolution, data contracts

- Docker, Linux fundamentals, Git/CI/CD

- Graph data modelling basics (Cypher, Neo4j/AGE)

Deliverables

- Source-system profiling reports

- Data-ingestion and transformation pipelines

- Data-quality rules and scorecards

- Entity-resolution logic

- Graph-loading pipelines

- Data lineage documentation

- Offline/air-gapped data movement processes

Preferred domain experience Experience in one or more of the following would be valuable:

- Power and utilities

- Smart metering

- SCADA or industrial IoT

- Energy accounting

- Fraud or leakage detection

- Banking and financial-risk analytics

- Telecom network analytics

- Manufacturing and asset intelligence

- Government or critical-infrastructure systems Domain experience should be preferred, not mandatory. A strong data skilled can learn utility terminology with help from the FDE and domain SME.

📌 Data Engineer (Hyderabad)
🏢 Nxtwave Disruptive Technologies (Hiring for a client)
📍 Hyderabad