Job Requirements

We are looking for a skilled Development Engineer – CDF Ingestion to design, develop, and maintain scalable data ingestion pipelines for collecting, validating, transforming, and loading data from multiple source systems into a centralized data platform.

The candidate will be responsible for building reliable batch and near-real-time ingestion workflows, handling structured and semi-structured datasets, implementing data quality checks, and ensuring data availability for downstream analytics and applications

Design, develop, and maintain data ingestion pipelines for CDF datasets from multiple source systems.

Implement batch and incremental data ingestion workflows to collect and load data into the centralized data platform

Develop data extraction and transformation logic using Python, SQL,



and relevant data processing frameworks.

Integrate data from REST APIs, databases, files, and external systems.

Implement schema validation, data type checks, record-level validation, and source-to-target reconciliation.

Handle missing records, duplicate data, malformed files, schema changes, and ingestion failures.

Develop mechanisms for error handling, retry logic, checkpointing, and recovery of failed ingestion jobs.

Apache Spark, AWS Glue, Athena, Step Functions, S3, multi-vendor schema ingestion (Ericsson/Oracle), datastore tuning

Work Experience

5-7Years

📌 Lead Software Engineer (Bengaluru)
🏢 Virtusa
📍 Bengaluru