Responsibilities-

Develop and test real-time data streaming pipelines using Apache Kafka, Apache Spark Streaming, or Apache Flink.

Validate streaming data for accuracy, completeness, consistency, and timeliness.

Design and execute test cases for batch and streaming data pipelines.

Write SQL and Python scripts for data validation and automated testing.

Monitor data pipelines, identify failures, and troubleshoot production issues.

Perform ETL and end-to-end testing for data ingestion workflows.

Verify schema changes and ensure data quality across systems.

Collaborate with data engineers, developers, QA teams, and business stakeholders.

Automate testing and integrate it into CI/CD pipelines.

Required Skills-

Apache Kafka

Apache Spark (Spark Streaming or Structured Streaming)

SQL

Python

ETL Testing

Data Validation

Linux/Unix

Git

REST APIs

JSON, Avro, or Parquet data formats

Basic cloud knowledge (AWS, Azure, or GCP)

Preferred Skills

Apache Flink

Kafka Connect

Airflow

Docker and Kubernetes

Jenkins or Azure DevOps

Monitoring tools (Grafana, Prometheus)

📌 Data Streaming And Testing Engineer Immediate Gurugram
🏢 Kezan consulting
📍 Gurugram