Location: Pan India

Experience: 5–7 Years

About The Role

We are looking for an experienced Python and PySpark Developer with solid hands-on expertise in PySpark ETL, Python, Spark SQL, DataFrames, and Delta Lake. The role focuses on developing, enhancing, testing, and supporting data processing pipelines, with an emphasis on using GenAI tools to improve development, debugging, testing, documentation, and code quality.

Job Description

The role involves maintaining and enhancing PySpark ETL jobs and troubleshooting data processing using Spark DataFrames, Spark SQL, and Delta Lake. You will work with file ingestion processes, including CSV, XML, and Hyper extracts, along with reporting-date handling and SQL-to-PySpark integration.

You will also develop and execute PyTest-based unit and integration tests and troubleshoot production issues using Python logging and exception handling. The position requires effective use of AI coding assistants such as GitHub Copilot, ChatGPT, or similar tools for code development, analysis, testing, debugging, documentation, and code reviews.





The role also emphasizes responsible AI usage, including data privacy, secure handling of production data, validation of AI-generated outputs, and awareness of prompt engineering for technical tasks.

Key Responsibilities

Maintain and enhance PySpark ETL jobs and data processing pipelines
Develop and troubleshoot Spark DataFrame, Spark SQL, and Delta Lake processing
Implement file ingestion for CSV, XML, and Hyper extracts
Work with SQL-to-PySpark integration and reporting-date handling
Develop and execute PyTest-based unit and integration tests
Troubleshoot production issues using Python logging and exception handling
Use AI coding assistants to accelerate development, debugging, testing, code review, and documentation
Analyze legacy code, perform impact assessments, and support root-cause analysis using AI-assisted techniques

Required Skills

Python
PySpark
Spark DataFrames and Spark SQL

📌 Contractor - Python & PySpark Developer (Hyderabad)
🏢 Vivantify Technology Solutions India
📍 Hyderabad