Job Title: Data Engineer - Python | PySpark

Experience: 6+ years

Job Type: India, Full-Time, Remote

Timings: 4pm-1am IST

Innovya Technologies is a fast‑growing software consulting firm specializing in business automation and up-to-date digital solutions. We enable organizations to accelerate value realization from their technology and transformation initiatives through expert, context‑driven product development and high‑quality engineering services.

About the Role:

We are looking for an experienced PySpark Developer with deep expertise data streaming development and deployment to join our dynamic team. A Python developer who will play a key role in maintaining, enhancing, and modernizing critical high volume and real time data pipelines. The ideal candidate will have a strong background in building scalable, fault-tolerant, and high-performance ELT applications, with a keen interest in AI enabled development.

You will work closely with cross-functional teams to design, analyse, build and deploy complex systems, providing technical direction and expertise to ensure the delivery of robust, efficient, and scalable solutions.

Key Responsibilities:

- Design, develop, and optimize scalable data pipelines using PySpark
- Process and analyze large datasets in distributed environments
- Collaborate with data engineers, analysts, and stakeholders to deliver data solutions
- Write efficient, reusable, and reliable PySpark code
- Perform data cleansing, transformation, and validation
- Optimize Spark jobs for performance and cost efficiency
- Integrate data from various sources (databases, APIs,



streaming platforms, etc.)
- Troubleshoot and debug data pipeline issues
- Maintain documentation for data workflows and processes

Required Skills:

- 6+ years of total experience in Python development, with at least 5 years in Apache Spark (PySpark).
- Hands-on experience with Python and PySpark in production data engineering environments.
- Use Spark DataFrames for large-scale data transformation and processing.
- Experience with complex aggregations, window functions, conditional aggregations, and performance tuning in Spark.
- Good understanding of Spark execution concepts such as partitions, shuffles, join behavior, and optimization techniques.
- Experience with API integrations, including pagination, request handling, response transformation, and fault tolerance.
- Working knowledge of AWS, preferably with services such as S3, Lambda, Glue, EMR, Athena, Step Functions, or similar.
- Experience handling JSON and schema-based parsing in distributed data pipelines.
- Familiarity with SQL-based data extraction and transformation.
- Ability to write production-quality, maintainable, and testable code.
- Good analytical and problem-solving skills, with the ability to understand complex data flows and business rules.
- Experience identifying bugs, troubleshooting processing issues, performing root cause analysis, and implementing fixes in distributed systems.
- Ability to create and maintain unit tests to improve code quality and reduce regressions.
- Preferred - 2+ years of DataBricks and Data Lake development experience

📌 ?Immediate Joiner: Data Engineer - Python | PySpark (6+ yrs)? (India)
🏢 Innovya Technologies
📍 India