01 Aug
|
Nameless
|
Hyderabad
01 Aug
Nameless
Hyderabad
Job Title: PySpark Data Engineer
Experience: 4+ Years
Location: Hyderabad
Job Summary:
We are seeking a Senior Spark Engineer to design and implement high -performance Spark execution patterns inside xFlows, supporting batch and streaming pipelines with built -in data quality, observability, and governance.
Requirements
Key Responsibilities
- Design and implement Spark -based execution frameworks for xFlows pipelines.
- Build reusable Spark components for:
- Readers (JDBC, Files, Kafka, CDC)
- Transformers (Join, Filter, Aggregate, Window, Union)
- Writers (Iceberg, Delta, Parquet, Snowflake)
- Optimize Spark performance (partitioning, caching, shuffles, memory).
- Implement Data Quality & Reconciliation execution patterns.
- Handle schema evolution, CDC, watermarking, and checkpoints.
- Integrate Spark jobs with EMR Serverless / Data bricks / Kubernetes.
- Publish execution metrics, logs, and lineage for observability.
- Work closely with platform & UI teams to support no -code execution.
Required Skills
- 4+ years of experience with Apache Spark
- Solid expertise in PySpark (preferred) or Spark Scala
- Deep understanding of Spark internals (DAGs, stages, shuffles, caching)
- Experience with Iceberg / Delta Lake
- Strong SQL skills
- Experience with batch and streaming pipeline