- We are looking for an experienced Python PySpark Developer to design, develop, and optimize large-scale data processing systems.
- The ideal candidate will work on big data platforms, build scalable ETL pipelines, and process high-volume datasets using Spark and Python.
- Develop and maintain data pipelines using Python and PySpark.
- Process and transform large datasets in distributed environments.
- Build scalable ETL/ELT workflows.
- Work with Apache Spark (PySpark) for batch and real-time processing.
- Optimize Spark jobs for performance and efficiency.
- Handle structured and unstructured data.
- Ingest data from multiple sources: Databases (SQL/NoSQL), APIs, Files (CSV, JSON, Parquet).
- Integrate with data platforms like Hadoop (HDFS)
and Cloud (AWS, Azure, GCP).
- Tune Spark jobs (partitioning, caching, parallelism).
- Optimize SQL queries and transformations.
- Improve data processing efficiency and cost.
- Collaborate with data engineers, data scientists, and analysts.
- Translate business requirements into technical solutions.
- Participate in code reviews and agile development practices.
- Debug and resolve issues in data pipelines.
- Monitor job execution and data quality.
- Ensure reliability and availability of data workflows.
Technical and Qualified Requirements
- Primary skills: Python, PySpark
Preferred Skills
- OpenSystem
- Python
- PySpark
- Big Data
- Data Processing