Required Skills:
- PySpark: In-depth knowledge of PySpark, Apache Spark, and Python.
- Data Processing: Solid understanding of data processing concepts, including data ingestion, data transformation, and data storage.
- Cloud Experience: Experience with cloud-based data platforms, including AWS, Azure, or Google Cloud.
- Expertise in DataFrames & Spark SQL, Spark Streaming with Apache Kafka real-time data ingestion pipelines