Job Title: Data Engineer - PySpark Developer (Core Spark Expert)

Total Experience: 5 to 8 Years

Relevant Experience: 4+ Years

Location: Remote

Rate: Open

Role Overview

We re looking for a strong PySpark Developer with deep expertise in core Apache Spark internals and hands-on experience modernizing legacy Spark codebases. The ideal candidate should be highly proficient in converting RDD-based pipelines into optimized DataFrame/Dataset-based architectures,

improving performance, scalability, and maintainability.

Key Responsibilities

Design, develop, and optimize scalable data pipelines using PySpark

Refactor and migrate legacy RDD-based Spark jobs to DataFrame/Dataset APIs

Analyze existing Spark workloads and improve performance (memory, compute, I/O)

Implement best practices for Spark optimization (partitioning, caching, joins, skew handling)

Work closely with data engineers, architects, and business teams to translate requirements into technical solutions

Debug and troubleshoot Spark jobs in distributed environments

Ensure data quality, reliability, and consistency across pipelines

Contribute to reusable frameworks, coding standards, and engineering best practices

Required Skills & Qualifications





4 8 years of experience in data engineering or big data development

Robust expertise in PySpark and core Spark concepts (RDDs, DataFrames, DAG, lazy evaluation,

shuffling)

Strong understanding of converting RDD-based implementations to DataFrame/Dataset APIs

Deep understanding of Spark performance tuning techniques

Solid programming skills in Python

Experience working with distributed data processing systems

Strong knowledge of SQL and data modeling concepts

Familiarity with data storage formats like Parquet, ORC, Avro

Preferred Qualifications

Experience with cloud platforms (Preferably GCP)

Hands-on experience with Spark on platforms like Databricks or EMR

Exposure to workflow orchestration tools (Airflow, etc.)

Knowledge of CI/CD pipelines and version control (Git)

Understanding of streaming frameworks (Spark Streaming / Structured Streaming

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Data Engineer - PySpark Developer (Noida)
🏢 Sparix Global
📍 Noida