- We are looking for an experienced Senior PySpark Developer with 8 years of experience in Data Engineering and Big Data technologies
- The ideal candidate should possess strong expertise in PySpark Python Spark SQL ETL development Data Warehousing and Cloud platforms AWS Azure GCP
- The candidate will be responsible for designing developing and optimizing large scale data processing solutions
Key Responsibilities:
- Key Responsibilities
- Design and develop scalable data pipelines using PySpark and Apache Spark
- Build and optimize ETL ELT frameworks for processing large volumes of structured and unstructured data
- Develop high performance Spark applications using Spark SQL DataFrames and RDDs
- Work closely with data architects and business teams to define data requirements
- Implement data quality validation and governance processes
- Optimize Spark jobs for performance memory management and scalability
- Integrate data from various sources including databases APIs cloud storage and streaming platforms
- Support production deployments and resolve performance bottlenecks
- Mentor junior developers and participate in code reviews
Technical Requirements:
- Mandatory Skills
- 8 years of IT experience
- Strong hands on experience in PySpark and Python
- Expertise in Apache Spark Spark SQL DataFrames and ETL Development
- Robust SQL and database concepts
- Experience with Hadoop Ecosystem Hive HDFS YARN
- Experience in Data Warehousing and Data Modeling
- Hands on experience with Unix Linux