Responsibilities:
• Design, develop, and optimize data pipelines using Apache Spark on Databricks.
• Implement ETL/ELT workflows to ingest, transform, and load data from various sources.
• Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions.
• Ensure data quality, integrity, and governance across all pipelines.
• Monitor and troubleshoot performance issues in Databricks notebooks and jobs.
• Integrate Databricks with cloud services (Azure Data Lake) and enterprise data platforms.
• Participate in code reviews, documentation, and agile ceremonies. Qualification and
Experience:
• 5+ years of experience in data engineering or big data development.
• Robust proficiency in Apache Spark, PySpark, and SQL.
• Hands-on experience with Databricks platform (including notebooks, jobs, clusters).
• Experience with cloud platforms (Azure) and cloud-native data services.
• Familiarity with Delta Lake, MLflow, and Databricks Workflows.
• Knowledge of CI/CD practices and version control (Git).
• Strong problem-solving and communication skills.