Roles & Responsibilities:
Develop and maintain data pipelines and workflows using PySpark on Databricks
Write Python scripts for data processing, transformation, and integration with Spark
Utilize PySpark and related libraries (e.g., pyspark-ai) to incorporate AI and machine learning functionalities into Spark workflows
Perform data analysis, transformations, and aggregations on large datasets
Deploy and optimize machine learning models within the Databricks setting
Collaborate with data engineers, data scientists, and business teams to understand requirements and deliver solutions
Use the Databricks platform to manage, monitor, and optimize Spark jobs and workflows
Work with the English SDK for Apache Spark to facilitate natural language interactions with Spark DataFrames
Ensure code quality, performance, and scalability of data pipelines and Spark jobs.
📌 Databricks Pune
🏢 ElfOnze Technologies
📍 Pune
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.