Job Description:

Job Summary: PySpark Developer (with Python Migration Focus) - to design, develop, and optimize big data solutions, with a critical focus on migrating and modernizing existing Python data processing codebases within a Health Insurance Company.

The PySpark Developer will be a key member of the Data Engineering team, responsible for leveraging PySpark and Apache Spark's distributed computing power to build scalable and high-performance data pipelines. A primary focus will be on migrating existing, complex Python scripts and data logic (e.g., Pandas, proprietary ETL tools) to the PySpark framework to enhance performance, scalability, and integration with the cloud-based data ecosystem. This role is vital for modernizing our data platform, handling massive volumes of sensitive health and claims data, and ensuring compliance with industry standards like HIPAA.

Key Responsibilities

PySpark Development & Data Engineering
Design and Develop Data Pipelines: Create, maintain, and optimize scalable Extract, Transform, Load (ETL)



and ELT pipelines using PySpark on a distributed computing setting (e.g., Databricks, AWS EMR, Azure Synapse).
Code Migration and Modernization: Lead the effort to re-engineer existing Python-based data processes, functions, and analytical logic (including legacy systems or complex Pandas transformations) into effective and performant PySpark code.
Performance Tuning: Profile, optimize, and fine-tune PySpark jobs for maximum speed and efficiency, focusing on techniques like partitioning, caching, broadcast variables, and query optimization to handle terabyte-scale healthcare datasets.
Data Quality and Governance: Implement robust data validation, cleansing, and monitoring procedures within PySpark jobs to ensure the highest quality and integrity of claims, member, and provider data.
Data Modeling: Collaborate with Data Architects to implement and optimize data models (Star, Snowflake, Data Vault) using technologies li

📌 Pyspark Developer Noida
🏢 Mancraft Consulting
📍 Noida