Required Skills:
PySpark: In-depth knowledge of PySpark, Apache Spark, and Python.
Data Processing: Solid understanding of data processing concepts, including data ingestion, data transformation, and data storage.
Cloud Experience: Experience with cloud-based data platforms, including AWS, Azure, or Google Cloud.
Expertise in DataFrames & Spark SQL, Spark Streaming with Apache Kafka real-time data ingestion pipelines