We are looking for an experienced Senior PySpark Developer with 8+ years of experience in Data Engineering and Big Data technologies. The ideal candidate should possess strong expertise in PySpark, Python, Spark SQL, ETL development, Data Warehousing, and Cloud platforms (AWS/Azure/GCP). The candidate will be responsible for designing, developing, and optimizing large-scale data processing solutions.
Responsibilities
- Design and develop scalable data pipelines using PySpark and Apache Spark.
- Build and optimize ETL/ELT frameworks for processing large volumes of structured and unstructured data.
- Develop high-performance Spark applications using Spark SQL, DataFrames, and RDDs.
- Work closely with data architects and business teams to define data requirements.
- Implement data quality, validation, and governance processes.
- Optimize Spark jobs for performance, memory management, and scalability.
- Integrate data from various sources including databases, APIs, cloud storage, and streaming platforms.
- Support production deployments and resolve performance bottlenecks.
- Mentor junior developers and participate in code reviews.
Technical Requirements Mandatory Skills
- 8+ years of IT experience.
- Strong hands-on experience in PySpark and Python.
- Expertise in Apache Spark, Spark SQL, DataFrames, and ETL Development.
- Robust SQL and database concepts.
- Experience with Hadoop Ecosystem (Hive, HDFS, YARN).
- Experience in Data Warehousing and Data Modeling.
- Hands-on experience with Unix/Linux.